blob: 4e080967898cb8ea196e7cfb2923979b2968bdfd [file] [log] [blame]
Andrew Boie08ce5a52016-02-22 13:28:10 -08001#!/usr/bin/env python3
Anas Nashifa792a3d2017-04-04 18:47:49 -04002# vim: set syntax=python ts=4 :
Anas Nashif3ae52622019-04-06 09:08:09 -04003# SPDX-License-Identifier: Apache-2.0
Andrew Boie6acbe632015-07-17 12:03:52 -07004"""Zephyr Sanity Tests
5
Marc Herberte5cedca2019-04-08 14:02:34 -07006Also check the "User and Developer Guides" at https://docs.zephyrproject.org/
7
Andrew Boie6acbe632015-07-17 12:03:52 -07008This script scans for the set of unit test applications in the git
9repository and attempts to execute them. By default, it tries to
10build each test case on one platform per architecture, using a precedence
Paul Sokolovskyff70add2017-06-16 01:31:54 +030011list defined in an architecture configuration file, and if possible
Anas Nashif83fc06a2019-06-22 11:04:10 -040012run the tests in any available emulators or simulators on the system.
Andrew Boie6acbe632015-07-17 12:03:52 -070013
Anas Nashifa792a3d2017-04-04 18:47:49 -040014Test cases are detected by the presence of a 'testcase.yaml' or a sample.yaml
15files in the application's project directory. This file may contain one or more
16blocks, each identifying a test scenario. The title of the block is a name for
17the test case, which only needs to be unique for the test cases specified in
18that testcase meta-data. The full canonical name for each test case is <path to
19test case>/<block>.
Andrew Boie6acbe632015-07-17 12:03:52 -070020
Anas Nashif3ba1d432017-12-05 15:28:44 -050021Each test block in the testcase meta data can define the following key/value
22pairs:
Andrew Boie6acbe632015-07-17 12:03:52 -070023
Anas Nashiffa695d22017-10-04 16:14:27 -040024 tags: <list of tags> (required)
Andrew Boie6acbe632015-07-17 12:03:52 -070025 A set of string tags for the testcase. Usually pertains to
26 functional domains but can be anything. Command line invocations
27 of this script can filter the set of tests to run based on tag.
28
Anas Nashifa792a3d2017-04-04 18:47:49 -040029 skip: <True|False> (default False)
Anas Nashif2bd99bc2015-10-12 13:10:57 -040030 skip testcase unconditionally. This can be used for broken tests.
31
Anas Nashifa792a3d2017-04-04 18:47:49 -040032 slow: <True|False> (default False)
Andy Rossdc4151f2019-01-03 14:17:43 -080033 Don't build or run this test case unless --enable-slow was passed
34 in on the command line. Intended for time-consuming test cases
35 that are only run under certain circumstances, like daily
36 builds.
Andrew Boie6bb087c2016-02-10 13:39:00 -080037
Anas Nashifa792a3d2017-04-04 18:47:49 -040038 extra_args: <list of extra arguments>
Sebastian Bøec2182612017-11-09 12:25:02 +010039 Extra cache entries to pass to CMake when building or running the
Andrew Boie6acbe632015-07-17 12:03:52 -070040 test case.
41
Anas Nashifebc329d2017-10-17 09:00:33 -040042 extra_configs: <list of extra configurations>
43 Extra configuration options to be merged with a master prj.conf
44 when building or running the test case.
45
Anas Nashifa792a3d2017-04-04 18:47:49 -040046 build_only: <True|False> (default False)
Marc Herberte5cedca2019-04-08 14:02:34 -070047 If true, don't try to run the test even if the selected platform
48 supports it.
Andrew Boie6acbe632015-07-17 12:03:52 -070049
Anas Nashifa792a3d2017-04-04 18:47:49 -040050 build_on_all: <True|False> (default False)
51 If true, attempt to build test on all available platforms.
52
53 depends_on: <list of features>
54 A board or platform can announce what features it supports, this option
55 will enable the test only those platforms that provide this feature.
56
57 min_ram: <integer>
58 minimum amount of RAM needed for this test to build and run. This is
59 compared with information provided by the board metadata.
60
61 min_flash: <integer>
62 minimum amount of ROM needed for this test to build and run. This is
63 compared with information provided by the board metadata.
64
65 timeout: <number of seconds>
Anas Nashif83fc06a2019-06-22 11:04:10 -040066 Length of time to run test in emulator before automatically killing it.
Andrew Boie6acbe632015-07-17 12:03:52 -070067 Default to 60 seconds.
68
Anas Nashifa792a3d2017-04-04 18:47:49 -040069 arch_whitelist: <list of arches, such as x86, arm, arc>
Andrew Boie6acbe632015-07-17 12:03:52 -070070 Set of architectures that this test case should only be run for.
71
Anas Nashifa792a3d2017-04-04 18:47:49 -040072 arch_exclude: <list of arches, such as x86, arm, arc>
Anas Nashif30d13872015-10-05 10:02:45 -040073 Set of architectures that this test case should not run on.
74
Anas Nashifa792a3d2017-04-04 18:47:49 -040075 platform_whitelist: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040076 Set of platforms that this test case should only be run for.
77
Anas Nashifa792a3d2017-04-04 18:47:49 -040078 platform_exclude: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040079 Set of platforms that this test case should not run on.
Andrew Boie6acbe632015-07-17 12:03:52 -070080
Anas Nashifa792a3d2017-04-04 18:47:49 -040081 extra_sections: <list of extra binary sections>
Andrew Boie52fef672016-11-29 12:21:59 -080082 When computing sizes, sanitycheck will report errors if it finds
83 extra, unexpected sections in the Zephyr binary unless they are named
84 here. They will not be included in the size calculation.
85
Anas Nashifa792a3d2017-04-04 18:47:49 -040086 filter: <expression>
Andrew Boie3ea78922016-03-24 14:46:00 -070087 Filter whether the testcase should be run by evaluating an expression
88 against an environment containing the following values:
89
90 { ARCH : <architecture>,
91 PLATFORM : <platform>,
Javier B Perez79414542016-08-08 12:24:59 -050092 <all CONFIG_* key/value pairs in the test's generated defconfig>,
Anas Nashif45a97862019-01-09 08:46:42 -050093 <all DT_* key/value pairs in the test's generated device tree file>,
94 <all CMake key/value pairs in the test's generated CMakeCache.txt file>,
Javier B Perez79414542016-08-08 12:24:59 -050095 *<env>: any environment variable available
Andrew Boie3ea78922016-03-24 14:46:00 -070096 }
97
98 The grammar for the expression language is as follows:
99
100 expression ::= expression "and" expression
101 | expression "or" expression
102 | "not" expression
103 | "(" expression ")"
104 | symbol "==" constant
105 | symbol "!=" constant
106 | symbol "<" number
107 | symbol ">" number
108 | symbol ">=" number
109 | symbol "<=" number
110 | symbol "in" list
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700111 | symbol ":" string
Andrew Boie3ea78922016-03-24 14:46:00 -0700112 | symbol
113
114 list ::= "[" list_contents "]"
115
116 list_contents ::= constant
117 | list_contents "," constant
118
119 constant ::= number
120 | string
121
122
123 For the case where expression ::= symbol, it evaluates to true
124 if the symbol is defined to a non-empty string.
125
126 Operator precedence, starting from lowest to highest:
127
128 or (left associative)
129 and (left associative)
130 not (right associative)
131 all comparison operators (non-associative)
132
133 arch_whitelist, arch_exclude, platform_whitelist, platform_exclude
134 are all syntactic sugar for these expressions. For instance
135
136 arch_exclude = x86 arc
137
138 Is the same as:
139
140 filter = not ARCH in ["x86", "arc"]
Andrew Boie6acbe632015-07-17 12:03:52 -0700141
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700142 The ':' operator compiles the string argument as a regular expression,
143 and then returns a true value only if the symbol's value in the environment
Anas Nashif578ae402019-07-12 07:54:35 -0700144 matches. For example, if CONFIG_SOC="stm32f107xc" then
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700145
Anas Nashif578ae402019-07-12 07:54:35 -0700146 filter = CONFIG_SOC : "stm.*"
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700147
148 Would match it.
149
Anas Nashifa792a3d2017-04-04 18:47:49 -0400150The set of test cases that actually run depends on directives in the testcase
151filed and options passed in on the command line. If there is any confusion,
152running with -v or --discard-report can help show why particular test cases
153were skipped.
Andrew Boie6acbe632015-07-17 12:03:52 -0700154
155Metrics (such as pass/fail state and binary size) for the last code
156release are stored in scripts/sanity_chk/sanity_last_release.csv.
157To update this, pass the --all --release options.
158
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -0500159To load arguments from a file, write '+' before the file name, e.g.,
160+file_name. File content must be one or more valid arguments separated by
161line break instead of white spaces.
162
Andrew Boie6acbe632015-07-17 12:03:52 -0700163Most everyday users will run with no arguments.
Andy Rossdc4151f2019-01-03 14:17:43 -0800164
Andrew Boie6acbe632015-07-17 12:03:52 -0700165"""
166
Sebastian Bøe56d74712019-01-21 15:48:46 +0100167import os
Sebastian Bøe56d74712019-01-21 15:48:46 +0100168if os.name == 'nt':
169 print("Running sanitycheck on Windows is not supported yet.")
170 print("https://github.com/zephyrproject-rtos/zephyr/issues/2664")
171 exit(1)
172
Anas Nashifaae71d72018-04-21 22:26:48 -0500173import contextlib
Anas Nashifa4c368e2018-10-15 09:45:59 -0400174import string
Anas Nashifaae71d72018-04-21 22:26:48 -0500175import mmap
Andrew Boie6acbe632015-07-17 12:03:52 -0700176import argparse
Andrew Boie6acbe632015-07-17 12:03:52 -0700177import sys
Andrew Boie6acbe632015-07-17 12:03:52 -0700178import re
Andrew Boie6acbe632015-07-17 12:03:52 -0700179import subprocess
180import multiprocessing
181import select
182import shutil
Marc Herbertaf1090c2019-04-30 14:11:29 -0700183import shlex
Andrew Boie6acbe632015-07-17 12:03:52 -0700184import signal
185import threading
Anas Nashif83fc06a2019-06-22 11:04:10 -0400186import concurrent.futures
187from threading import BoundedSemaphore
188import queue
Andrew Boie6acbe632015-07-17 12:03:52 -0700189import time
Anas Nashif654ec5982019-04-11 08:38:21 -0400190import datetime
Andrew Boie6acbe632015-07-17 12:03:52 -0700191import csv
Anas Nashif83fc06a2019-06-22 11:04:10 -0400192import yaml
Andrew Boie5d4eb782015-10-02 10:04:56 -0700193import glob
Anas Nashif73440ea2018-02-19 10:57:03 -0600194import serial
Daniel Leung6b170072016-04-07 12:10:25 -0700195import concurrent
Anas Nashifb3311ed2017-04-13 14:44:48 -0400196import xml.etree.ElementTree as ET
Anas Nashif035799f2017-05-13 21:31:53 -0400197from collections import OrderedDict
198from itertools import islice
Anas Nashife24350c2018-07-11 15:09:22 -0500199from pathlib import Path
200from distutils.spawn import find_executable
Andrew Boie6acbe632015-07-17 12:03:52 -0700201
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700202import logging
Anas Nashif83fc06a2019-06-22 11:04:10 -0400203
204
205hw_map_local = threading.Lock()
206
Anas Nashif3ba1d432017-12-05 15:28:44 -0500207
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700208log_format = "%(levelname)s %(name)s::%(module)s.%(funcName)s():%(lineno)d: %(message)s"
Anas Nashif3ba1d432017-12-05 15:28:44 -0500209logging.basicConfig(format=log_format, level=30)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700210
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +0300211ZEPHYR_BASE = os.environ.get("ZEPHYR_BASE")
212if not ZEPHYR_BASE:
Anas Nashif427cdd32015-08-06 07:25:42 -0400213 sys.stderr.write("$ZEPHYR_BASE environment variable undefined.\n")
Andrew Boie6acbe632015-07-17 12:03:52 -0700214 exit(1)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700215
Marc Herbert1c8632c2019-04-15 17:58:45 -0700216# Use this for internal comparisons; that's what canonicalization is
217# for. Don't use it when invoking other components of the build system
218# to avoid confusing and hard to trace inconsistencies in error messages
219# and logs, generated Makefiles, etc. compared to when users invoke these
220# components directly.
221# Note "normalization" is different from canonicalization, see os.path.
222canonical_zephyr_base = os.path.realpath(ZEPHYR_BASE)
223
Andrew Boie3ea78922016-03-24 14:46:00 -0700224sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/"))
225
Anas Nashif83fc06a2019-06-22 11:04:10 -0400226from sanity_chk import scl
227from sanity_chk import expr_parser
228
Andrew Boie3ea78922016-03-24 14:46:00 -0700229
Andrew Boie6acbe632015-07-17 12:03:52 -0700230VERBOSE = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -0400231
Andrew Boie6acbe632015-07-17 12:03:52 -0700232RELEASE_DATA = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk",
233 "sanity_last_release.csv")
Andrew Boie6acbe632015-07-17 12:03:52 -0700234
235if os.isatty(sys.stdout.fileno()):
236 TERMINAL = True
237 COLOR_NORMAL = '\033[0m'
238 COLOR_RED = '\033[91m'
239 COLOR_GREEN = '\033[92m'
240 COLOR_YELLOW = '\033[93m'
241else:
242 TERMINAL = False
243 COLOR_NORMAL = ""
244 COLOR_RED = ""
245 COLOR_GREEN = ""
246 COLOR_YELLOW = ""
247
Anas Nashif45a97862019-01-09 08:46:42 -0500248class CMakeCacheEntry:
249 '''Represents a CMake cache entry.
250
251 This class understands the type system in a CMakeCache.txt, and
252 converts the following cache types to Python types:
253
254 Cache Type Python type
255 ---------- -------------------------------------------
256 FILEPATH str
257 PATH str
258 STRING str OR list of str (if ';' is in the value)
259 BOOL bool
260 INTERNAL str OR list of str (if ';' is in the value)
261 ---------- -------------------------------------------
262 '''
263
264 # Regular expression for a cache entry.
265 #
266 # CMake variable names can include escape characters, allowing a
267 # wider set of names than is easy to match with a regular
268 # expression. To be permissive here, use a non-greedy match up to
269 # the first colon (':'). This breaks if the variable name has a
270 # colon inside, but it's good enough.
271 CACHE_ENTRY = re.compile(
272 r'''(?P<name>.*?) # name
273 :(?P<type>FILEPATH|PATH|STRING|BOOL|INTERNAL) # type
274 =(?P<value>.*) # value
275 ''', re.X)
276
277 @classmethod
278 def _to_bool(cls, val):
279 # Convert a CMake BOOL string into a Python bool.
280 #
281 # "True if the constant is 1, ON, YES, TRUE, Y, or a
282 # non-zero number. False if the constant is 0, OFF, NO,
283 # FALSE, N, IGNORE, NOTFOUND, the empty string, or ends in
284 # the suffix -NOTFOUND. Named boolean constants are
285 # case-insensitive. If the argument is not one of these
286 # constants, it is treated as a variable."
287 #
288 # https://cmake.org/cmake/help/v3.0/command/if.html
289 val = val.upper()
290 if val in ('ON', 'YES', 'TRUE', 'Y'):
291 return 1
292 elif val in ('OFF', 'NO', 'FALSE', 'N', 'IGNORE', 'NOTFOUND', ''):
293 return 0
294 elif val.endswith('-NOTFOUND'):
295 return 0
296 else:
297 try:
298 v = int(val)
299 return v != 0
300 except ValueError as exc:
301 raise ValueError('invalid bool {}'.format(val)) from exc
302
303 @classmethod
304 def from_line(cls, line, line_no):
305 # Comments can only occur at the beginning of a line.
306 # (The value of an entry could contain a comment character).
307 if line.startswith('//') or line.startswith('#'):
308 return None
309
310 # Whitespace-only lines do not contain cache entries.
311 if not line.strip():
312 return None
313
314 m = cls.CACHE_ENTRY.match(line)
315 if not m:
316 return None
317
318 name, type_, value = (m.group(g) for g in ('name', 'type', 'value'))
319 if type_ == 'BOOL':
320 try:
321 value = cls._to_bool(value)
322 except ValueError as exc:
323 args = exc.args + ('on line {}: {}'.format(line_no, line),)
324 raise ValueError(args) from exc
Anas Nashif83fc06a2019-06-22 11:04:10 -0400325 elif type_ in ['STRING','INTERNAL']:
Anas Nashif45a97862019-01-09 08:46:42 -0500326 # If the value is a CMake list (i.e. is a string which
327 # contains a ';'), convert to a Python list.
328 if ';' in value:
329 value = value.split(';')
330
331 return CMakeCacheEntry(name, value)
332
333 def __init__(self, name, value):
334 self.name = name
335 self.value = value
336
337 def __str__(self):
338 fmt = 'CMakeCacheEntry(name={}, value={})'
339 return fmt.format(self.name, self.value)
340
341
342class CMakeCache:
343 '''Parses and represents a CMake cache file.'''
344
345 @staticmethod
346 def from_file(cache_file):
347 return CMakeCache(cache_file)
348
349 def __init__(self, cache_file):
350 self.cache_file = cache_file
351 self.load(cache_file)
352
353 def load(self, cache_file):
354 entries = []
355 with open(cache_file, 'r') as cache:
356 for line_no, line in enumerate(cache):
357 entry = CMakeCacheEntry.from_line(line, line_no)
358 if entry:
359 entries.append(entry)
360 self._entries = OrderedDict((e.name, e) for e in entries)
361
362 def get(self, name, default=None):
363 entry = self._entries.get(name)
364 if entry is not None:
365 return entry.value
366 else:
367 return default
368
369 def get_list(self, name, default=None):
370 if default is None:
371 default = []
372 entry = self._entries.get(name)
373 if entry is not None:
374 value = entry.value
375 if isinstance(value, list):
376 return value
377 elif isinstance(value, str):
378 return [value] if value else []
379 else:
380 msg = 'invalid value {} type {}'
381 raise RuntimeError(msg.format(value, type(value)))
382 else:
383 return default
384
385 def __contains__(self, name):
386 return name in self._entries
387
388 def __getitem__(self, name):
389 return self._entries[name].value
390
391 def __setitem__(self, name, entry):
392 if not isinstance(entry, CMakeCacheEntry):
393 msg = 'improper type {} for value {}, expecting CMakeCacheEntry'
394 raise TypeError(msg.format(type(entry), entry))
395 self._entries[name] = entry
396
397 def __delitem__(self, name):
398 del self._entries[name]
399
400 def __iter__(self):
401 return iter(self._entries.values())
402
Andrew Boie6acbe632015-07-17 12:03:52 -0700403class SanityCheckException(Exception):
404 pass
405
Anas Nashif3ba1d432017-12-05 15:28:44 -0500406
Andrew Boie6acbe632015-07-17 12:03:52 -0700407class SanityRuntimeError(SanityCheckException):
408 pass
409
Andrew Boie6acbe632015-07-17 12:03:52 -0700410class ConfigurationError(SanityCheckException):
411 def __init__(self, cfile, message):
Anas Nashif83fc06a2019-06-22 11:04:10 -0400412 SanityCheckException.__init__(self, cfile + ": " + message)
Andrew Boie6acbe632015-07-17 12:03:52 -0700413
Anas Nashif83fc06a2019-06-22 11:04:10 -0400414class BuildError(SanityCheckException):
Andrew Boie6acbe632015-07-17 12:03:52 -0700415 pass
416
Anas Nashif3ba1d432017-12-05 15:28:44 -0500417
Anas Nashif83fc06a2019-06-22 11:04:10 -0400418class ExecutionError(SanityCheckException):
Andrew Boie6acbe632015-07-17 12:03:52 -0700419 pass
420
Anas Nashif3ba1d432017-12-05 15:28:44 -0500421
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800422log_file = None
423
Andrew Boie6acbe632015-07-17 12:03:52 -0700424# Debug Functions
Anas Nashif654ec5982019-04-11 08:38:21 -0400425def info(what, show_time=True):
426 if options.timestamps and show_time:
427 date = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
428 what = "{}: {}".format(date, what)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800429 sys.stdout.write(what + "\n")
Paul Sokolovsky3ba08762017-10-27 14:53:24 +0300430 sys.stdout.flush()
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800431 if log_file:
432 log_file.write(what + "\n")
433 log_file.flush()
Andrew Boie6acbe632015-07-17 12:03:52 -0700434
Anas Nashif3ba1d432017-12-05 15:28:44 -0500435
Andrew Boie6acbe632015-07-17 12:03:52 -0700436def error(what):
Anas Nashif654ec5982019-04-11 08:38:21 -0400437 if options.timestamps:
438 date = datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
439 what = "{}: {}".format(date, what)
Andrew Boie6acbe632015-07-17 12:03:52 -0700440 sys.stderr.write(COLOR_RED + what + COLOR_NORMAL + "\n")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800441 if log_file:
442 log_file(what + "\n")
443 log_file.flush()
Andrew Boie6acbe632015-07-17 12:03:52 -0700444
Anas Nashif3ba1d432017-12-05 15:28:44 -0500445
Andrew Boie08ce5a52016-02-22 13:28:10 -0800446def debug(what):
447 if VERBOSE >= 1:
448 info(what)
449
Anas Nashif3ba1d432017-12-05 15:28:44 -0500450
Andrew Boie6acbe632015-07-17 12:03:52 -0700451def verbose(what):
452 if VERBOSE >= 2:
Andrew Boie08ce5a52016-02-22 13:28:10 -0800453 info(what)
Andrew Boie6acbe632015-07-17 12:03:52 -0700454
Anas Nashif576be982017-12-23 20:20:27 -0500455class HarnessImporter:
456
457 def __init__(self, name):
458 sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/sanity_chk"))
459 module = __import__("harness")
460 if name:
461 my_class = getattr(module, name)
462 else:
463 my_class = getattr(module, "Test")
464
465 self.instance = my_class()
Anas Nashif3ba1d432017-12-05 15:28:44 -0500466
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300467class Handler:
Anas Nashifd18ec532019-04-11 23:20:39 -0400468 def __init__(self, instance, type_str="build"):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300469 """Constructor
470
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300471 """
472 self.lock = threading.Lock()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400473
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300474 self.state = "waiting"
Anas Nashif13773752018-07-06 18:20:23 -0500475 self.run = False
Anas Nashif83fc06a2019-06-22 11:04:10 -0400476 self.duration = 0
Anas Nashifd18ec532019-04-11 23:20:39 -0400477 self.type_str = type_str
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300478
Anas Nashifdf7ee612018-07-07 06:09:01 -0500479 self.binary = None
Jan Kowalewski265895b2019-01-07 16:40:24 +0100480 self.pid_fn = None
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500481 self.call_make_run = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500482
Anas Nashifd3384fb2018-02-22 06:44:16 -0600483 self.name = instance.name
484 self.instance = instance
Anas Nashif83fc06a2019-06-22 11:04:10 -0400485 self.timeout = instance.testcase.timeout
486 self.sourcedir = instance.testcase.source_dir
487 self.build_dir = instance.build_dir
488 self.log = os.path.join(self.build_dir, "handler.log")
Anas Nashifd3384fb2018-02-22 06:44:16 -0600489 self.returncode = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -0400490 self.set_state("running", self.duration)
Anas Nashifd3384fb2018-02-22 06:44:16 -0600491
Anas Nashif83fc06a2019-06-22 11:04:10 -0400492 self.args = []
493
494 def set_state(self, state, duration):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300495 self.lock.acquire()
496 self.state = state
Anas Nashif83fc06a2019-06-22 11:04:10 -0400497 self.duration = duration
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300498 self.lock.release()
499
500 def get_state(self):
501 self.lock.acquire()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400502 ret = (self.state, self.duration)
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300503 self.lock.release()
504 return ret
505
Anas Nashif83fc06a2019-06-22 11:04:10 -0400506 def record(self, harness):
507 if harness.recording:
508 filename = os.path.join(options.outdir,
509 self.instance.platform.name,
510 self.instance.testcase.name, "recording.csv")
511 with open(filename, "at") as csvfile:
512 cw = csv.writer(csvfile, harness.fieldnames, lineterminator=os.linesep)
513 cw.writerow(harness.fieldnames)
514 for instance in harness.recording:
515 cw.writerow(instance)
516
Anas Nashifdf7ee612018-07-07 06:09:01 -0500517class BinaryHandler(Handler):
Anas Nashifd18ec532019-04-11 23:20:39 -0400518 def __init__(self, instance, type_str):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500519 """Constructor
520
521 @param instance Test Instance
522 """
Anas Nashifd18ec532019-04-11 23:20:39 -0400523 super().__init__(instance, type_str)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500524
525 self.valgrind = False
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100526 self.terminated = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500527
Jan Kowalewski265895b2019-01-07 16:40:24 +0100528 def try_kill_process_by_pid(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -0400529 if self.pid_fn:
Jan Kowalewski265895b2019-01-07 16:40:24 +0100530 pid = int(open(self.pid_fn).read())
531 os.unlink(self.pid_fn)
532 self.pid_fn = None # clear so we don't try to kill the binary twice
533 try:
534 os.kill(pid, signal.SIGTERM)
535 except ProcessLookupError:
536 pass
537
Anas Nashifdf7ee612018-07-07 06:09:01 -0500538 def _output_reader(self, proc, harness):
539 log_out_fp = open(self.log, "wt")
540 for line in iter(proc.stdout.readline, b''):
541 verbose("OUTPUT: {0}".format(line.decode('utf-8').rstrip()))
542 log_out_fp.write(line.decode('utf-8'))
543 log_out_fp.flush()
544 harness.handle(line.decode('utf-8').rstrip())
545 if harness.state:
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100546 try:
547 #POSIX arch based ztests end on their own,
548 #so let's give it up to 100ms to do so
549 proc.wait(0.1)
550 except subprocess.TimeoutExpired:
551 proc.terminate()
552 self.terminated = True
Anas Nashifdf7ee612018-07-07 06:09:01 -0500553 break
554
555 log_out_fp.close()
556
557 def handle(self):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500558
Anas Nashif83fc06a2019-06-22 11:04:10 -0400559 harness_name = self.instance.testcase.harness.capitalize()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500560 harness_import = HarnessImporter(harness_name)
561 harness = harness_import.instance
562 harness.configure(self.instance)
563
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500564 if self.call_make_run:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400565 command = [get_generator()[0], "run"]
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500566 else:
567 command = [self.binary]
Anas Nashifdf7ee612018-07-07 06:09:01 -0500568
Alberto Escolar Piedrasb91f3742019-07-02 10:06:22 +0200569 if self.valgrind and shutil.which("valgrind"):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500570 command = ["valgrind", "--error-exitcode=2",
Alberto Escolar Piedras3980e0c2018-12-12 17:41:04 +0100571 "--leak-check=full",
572 "--suppressions="+ZEPHYR_BASE+"/scripts/valgrind.supp",
Anas Nashif83fc06a2019-06-22 11:04:10 -0400573 "--log-file="+self.build_dir+"/valgrind.log"
Alberto Escolar Piedras3980e0c2018-12-12 17:41:04 +0100574 ] + command
Anas Nashifdf7ee612018-07-07 06:09:01 -0500575
Marc Herbertaf1090c2019-04-30 14:11:29 -0700576 verbose("Spawning process: " +
Jan Van Winkel6b9e1602019-01-03 19:18:35 +0100577 " ".join(shlex.quote(word) for word in command) + os.linesep +
Anas Nashif83fc06a2019-06-22 11:04:10 -0400578 "Spawning process in directory: " + self.build_dir)
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200579
Anas Nashif83fc06a2019-06-22 11:04:10 -0400580 start_time = time.time()
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200581
Anas Nashif83fc06a2019-06-22 11:04:10 -0400582 with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.build_dir) as proc:
Marc Herbertaf1090c2019-04-30 14:11:29 -0700583 verbose("Spawning BinaryHandler Thread for %s" % self.name)
Flavio Ceolin063ab902019-10-15 16:10:49 -0700584 t = threading.Thread(target=self._output_reader, args=(proc, harness, ), daemon=True)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500585 t.start()
586 t.join(self.timeout)
587 if t.is_alive():
Jan Kowalewski265895b2019-01-07 16:40:24 +0100588 self.try_kill_process_by_pid()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500589 proc.terminate()
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100590 self.terminated = True
Anas Nashifdf7ee612018-07-07 06:09:01 -0500591 t.join()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500592 proc.wait()
593 self.returncode = proc.returncode
Anas Nashifdf7ee612018-07-07 06:09:01 -0500594
Anas Nashif83fc06a2019-06-22 11:04:10 -0400595 handler_time = time.time() - start_time
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200596
Anas Nashifdf7ee612018-07-07 06:09:01 -0500597 if options.enable_coverage:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400598 subprocess.call(["GCOV_PREFIX=" + self.build_dir,
599 "gcov", self.sourcedir, "-b", "-s", self.build_dir], shell=True)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500600
Jan Kowalewski265895b2019-01-07 16:40:24 +0100601 self.try_kill_process_by_pid()
602
Anas Nashif83fc06a2019-06-22 11:04:10 -0400603 # FIXME: This is needed when killing the simulator, the console is
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500604 # garbled and needs to be reset. Did not find a better way to do that.
605
606 subprocess.call(["stty", "sane"])
Anas Nashifdf7ee612018-07-07 06:09:01 -0500607 self.instance.results = harness.tests
Anas Nashif83fc06a2019-06-22 11:04:10 -0400608 if not self.terminated and self.returncode != 0:
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100609 #When a process is killed, the default handler returns 128 + SIGTERM
610 #so in that case the return code itself is not meaningful
Anas Nashiff72d1902019-10-18 17:20:44 -0400611 self.set_state("failed", handler_time)
Anas Nashif83fc06a2019-06-22 11:04:10 -0400612 self.instance.reason = "Handler error"
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100613 elif harness.state:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400614 self.set_state(harness.state, handler_time)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500615 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400616 self.set_state("timeout", handler_time)
617 self.instance.reason = "Handler timeout"
618
619 self.record(harness)
Anas Nashif73440ea2018-02-19 10:57:03 -0600620
621class DeviceHandler(Handler):
622
Anas Nashifd18ec532019-04-11 23:20:39 -0400623 def __init__(self, instance, type_str):
Anas Nashif73440ea2018-02-19 10:57:03 -0600624 """Constructor
625
626 @param instance Test Instance
627 """
Anas Nashifd18ec532019-04-11 23:20:39 -0400628 super().__init__(instance, type_str)
Anas Nashif73440ea2018-02-19 10:57:03 -0600629
Anas Nashif83fc06a2019-06-22 11:04:10 -0400630 self.suite = None
631
Marti Bolivar5591ca22019-02-07 15:53:39 -0700632 def monitor_serial(self, ser, halt_fileno, harness):
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500633 log_out_fp = open(self.log, "wt")
Anas Nashif73440ea2018-02-19 10:57:03 -0600634
Marti Bolivar5591ca22019-02-07 15:53:39 -0700635 ser_fileno = ser.fileno()
636 readlist = [halt_fileno, ser_fileno]
637
Anas Nashif73440ea2018-02-19 10:57:03 -0600638 while ser.isOpen():
Marti Bolivar5591ca22019-02-07 15:53:39 -0700639 readable, _, _ = select.select(readlist, [], [], self.timeout)
640
641 if halt_fileno in readable:
642 verbose('halted')
643 ser.close()
644 break
645 if ser_fileno not in readable:
646 continue # Timeout.
647
648 serial_line = None
Anas Nashif61e21632018-04-08 13:30:16 -0500649 try:
650 serial_line = ser.readline()
651 except TypeError:
652 pass
Anas Nashif83fc06a2019-06-22 11:04:10 -0400653 except serial.SerialException:
Anas Nashif59237602019-03-03 10:36:35 -0500654 ser.close()
655 break
Anas Nashif61e21632018-04-08 13:30:16 -0500656
Marti Bolivar5591ca22019-02-07 15:53:39 -0700657 # Just because ser_fileno has data doesn't mean an entire line
658 # is available yet.
Anas Nashif73440ea2018-02-19 10:57:03 -0600659 if serial_line:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600660 sl = serial_line.decode('utf-8', 'ignore')
661 verbose("DEVICE: {0}".format(sl.rstrip()))
662
663 log_out_fp.write(sl)
664 log_out_fp.flush()
665 harness.handle(sl.rstrip())
Marti Bolivar5591ca22019-02-07 15:53:39 -0700666
Anas Nashif73440ea2018-02-19 10:57:03 -0600667 if harness.state:
668 ser.close()
669 break
670
671 log_out_fp.close()
672
Anas Nashif83fc06a2019-06-22 11:04:10 -0400673 def device_is_available(self, device):
674 for i in self.suite.connected_hardware:
675 if i['platform'] == device and i['available'] and i['connected']:
676 return True
677
678 return False
679
680 def get_available_device(self, device):
681 for i in self.suite.connected_hardware:
682 if i['platform'] == device and i['available']:
683 i['available'] = False
684 i['counter'] += 1
685 return i
686
687 return None
688
689 def make_device_available(self, serial):
690 with hw_map_local:
691 for i in self.suite.connected_hardware:
692 if i['serial'] == serial:
693 i['available'] = True
694
Anas Nashif73440ea2018-02-19 10:57:03 -0600695 def handle(self):
696 out_state = "failed"
697
Anas Nashif83fc06a2019-06-22 11:04:10 -0400698 if options.west_flash:
699 command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir]
700 if options.west_runner:
Michael Scott421ce462019-06-18 09:37:46 -0700701 command.append("--runner")
702 command.append(options.west_runner)
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200703 # There are three ways this option is used.
Andy Doan79c48842019-02-08 10:09:04 -0600704 # 1) bare: --west-flash
705 # This results in options.west_flash == []
706 # 2) with a value: --west-flash="--board-id=42"
707 # This results in options.west_flash == "--board-id=42"
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200708 # 3) Multiple values: --west-flash="--board-id=42,--erase"
709 # This results in options.west_flash == "--board-id=42 --erase"
Andy Doan79c48842019-02-08 10:09:04 -0600710 if options.west_flash != []:
711 command.append('--')
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200712 command.extend(options.west_flash.split(','))
Anas Nashifd3384fb2018-02-22 06:44:16 -0600713 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400714 command = [get_generator()[0], "-C", self.build_dir, "flash"]
Anas Nashifd3384fb2018-02-22 06:44:16 -0600715
Anas Nashifd3384fb2018-02-22 06:44:16 -0600716
Anas Nashif83fc06a2019-06-22 11:04:10 -0400717 while not self.device_is_available(self.instance.platform.name):
718 time.sleep(1)
719
720 hardware = self.get_available_device(self.instance.platform.name)
721
722 runner = hardware.get('runner', None)
723 if runner:
724 board_id = hardware.get("id", None)
725 product = hardware.get("product", None)
726 command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir]
727 command.append("--runner")
728 command.append(hardware.get('runner', None))
729 if runner == "pyocd":
730 command.append("--board-id")
731 command.append(board_id)
732 elif runner == "nrfjprog":
733 command.append('--')
734 command.append("--snr")
735 command.append(board_id)
736 elif runner == "openocd" and product == "STM32 STLink":
737 command.append('--')
738 command.append("--cmd-pre-init")
739 command.append("hla_serial %s" %(board_id))
740 elif runner == "openocd" and product == "EDBG CMSIS-DAP":
741 command.append('--')
742 command.append("--cmd-pre-init")
743 command.append("cmsis_dap_serial %s" %(board_id))
744 elif runner == "jlink":
745 command.append("--tool-opt=-SelectEmuBySN %s" %(board_id))
746
747 serial_device = hardware['serial']
748
749 try:
750 ser = serial.Serial(
751 serial_device,
752 baudrate=115200,
753 parity=serial.PARITY_NONE,
754 stopbits=serial.STOPBITS_ONE,
755 bytesize=serial.EIGHTBITS,
756 timeout=self.timeout
757 )
758 except serial.SerialException as e:
759 self.set_state("failed", 0)
760 error("Serial device err: %s" %(str(e)))
761 self.make_device_available(serial_device)
762 return
Anas Nashif73440ea2018-02-19 10:57:03 -0600763
764 ser.flush()
765
Anas Nashif83fc06a2019-06-22 11:04:10 -0400766 harness_name = self.instance.testcase.harness.capitalize()
Anas Nashif73440ea2018-02-19 10:57:03 -0600767 harness_import = HarnessImporter(harness_name)
768 harness = harness_import.instance
769 harness.configure(self.instance)
Anas Nashif83fc06a2019-06-22 11:04:10 -0400770 read_pipe, write_pipe = os.pipe()
771 start_time = time.time()
Anas Nashif73440ea2018-02-19 10:57:03 -0600772
Marti Bolivar5591ca22019-02-07 15:53:39 -0700773 t = threading.Thread(target=self.monitor_serial, daemon=True,
Anas Nashif83fc06a2019-06-22 11:04:10 -0400774 args=(ser, read_pipe, harness))
Anas Nashif73440ea2018-02-19 10:57:03 -0600775 t.start()
776
Andy Doan79c48842019-02-08 10:09:04 -0600777 logging.debug('Flash command: %s', command)
Anas Nashif61e21632018-04-08 13:30:16 -0500778 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400779 if VERBOSE and not runner:
Marti Bolivar303b5222019-02-07 15:50:55 -0700780 subprocess.check_call(command)
781 else:
782 subprocess.check_output(command, stderr=subprocess.PIPE)
Anas Nashif83fc06a2019-06-22 11:04:10 -0400783
Anas Nashif61e21632018-04-08 13:30:16 -0500784 except subprocess.CalledProcessError:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400785 os.write(write_pipe, b'x') # halt the thread
Anas Nashif73440ea2018-02-19 10:57:03 -0600786
787 t.join(self.timeout)
788 if t.is_alive():
789 out_state = "timeout"
Anas Nashif73440ea2018-02-19 10:57:03 -0600790
791 if ser.isOpen():
792 ser.close()
793
Anas Nashifd3384fb2018-02-22 06:44:16 -0600794 if out_state == "timeout":
Anas Nashif83fc06a2019-06-22 11:04:10 -0400795 for c in self.instance.testcase.cases:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600796 if c not in harness.tests:
797 harness.tests[c] = "BLOCK"
Anas Nashif61e21632018-04-08 13:30:16 -0500798
Anas Nashif83fc06a2019-06-22 11:04:10 -0400799
800 handler_time = time.time() - start_time
801
Anas Nashif61e21632018-04-08 13:30:16 -0500802 self.instance.results = harness.tests
Anas Nashif73440ea2018-02-19 10:57:03 -0600803 if harness.state:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400804 self.set_state(harness.state, handler_time)
Anas Nashif73440ea2018-02-19 10:57:03 -0600805 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400806 self.set_state(out_state, handler_time)
Anas Nashif73440ea2018-02-19 10:57:03 -0600807
Anas Nashif83fc06a2019-06-22 11:04:10 -0400808 self.make_device_available(serial_device)
Anas Nashif3ba1d432017-12-05 15:28:44 -0500809
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300810class QEMUHandler(Handler):
Andrew Boie6acbe632015-07-17 12:03:52 -0700811 """Spawns a thread to monitor QEMU output from pipes
812
Anas Nashif3ac7b3a2017-08-03 10:03:02 -0400813 We pass QEMU_PIPE to 'make run' and monitor the pipes for output.
Andrew Boie6acbe632015-07-17 12:03:52 -0700814 We need to do this as once qemu starts, it runs forever until killed.
815 Test cases emit special messages to the console as they run, we check
816 for these to collect whether the test passed or failed.
817 """
Andrew Boie6acbe632015-07-17 12:03:52 -0700818
Anas Nashif83fc06a2019-06-22 11:04:10 -0400819
820 def __init__(self, instance, type_str):
821 """Constructor
822
823 @param instance Test instance
824 """
825
826 super().__init__(instance, type_str)
827 self.fifo_fn = os.path.join(instance.build_dir, "qemu-fifo")
828
829 self.pid_fn = os.path.join(instance.build_dir, "qemu.pid")
830
831
Andrew Boie6acbe632015-07-17 12:03:52 -0700832 @staticmethod
Anas Nashif576be982017-12-23 20:20:27 -0500833 def _thread(handler, timeout, outdir, logfile, fifo_fn, pid_fn, results, harness):
Andrew Boie6acbe632015-07-17 12:03:52 -0700834 fifo_in = fifo_fn + ".in"
835 fifo_out = fifo_fn + ".out"
836
837 # These in/out nodes are named from QEMU's perspective, not ours
838 if os.path.exists(fifo_in):
839 os.unlink(fifo_in)
840 os.mkfifo(fifo_in)
841 if os.path.exists(fifo_out):
842 os.unlink(fifo_out)
843 os.mkfifo(fifo_out)
844
845 # We don't do anything with out_fp but we need to open it for
846 # writing so that QEMU doesn't block, due to the way pipes work
847 out_fp = open(fifo_in, "wb")
848 # Disable internal buffering, we don't
849 # want read() or poll() to ever block if there is data in there
850 in_fp = open(fifo_out, "rb", buffering=0)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800851 log_out_fp = open(logfile, "wt")
Andrew Boie6acbe632015-07-17 12:03:52 -0700852
853 start_time = time.time()
854 timeout_time = start_time + timeout
855 p = select.poll()
856 p.register(in_fp, select.POLLIN)
Anas Nashif39ae72b2018-08-29 01:45:38 -0400857 out_state = None
Andrew Boie6acbe632015-07-17 12:03:52 -0700858
Andrew Boie6acbe632015-07-17 12:03:52 -0700859 line = ""
Anas Nashif74dbe332019-01-17 17:11:37 -0500860 timeout_extended = False
Andrew Boie6acbe632015-07-17 12:03:52 -0700861 while True:
862 this_timeout = int((timeout_time - time.time()) * 1000)
863 if this_timeout < 0 or not p.poll(this_timeout):
Anas Nashif39ae72b2018-08-29 01:45:38 -0400864 if not out_state:
865 out_state = "timeout"
Andrew Boie6acbe632015-07-17 12:03:52 -0700866 break
867
Genaro Saucedo Tejada2968b362016-08-09 15:11:53 -0500868 try:
869 c = in_fp.read(1).decode("utf-8")
870 except UnicodeDecodeError:
871 # Test is writing something weird, fail
872 out_state = "unexpected byte"
873 break
874
Andrew Boie6acbe632015-07-17 12:03:52 -0700875 if c == "":
876 # EOF, this shouldn't happen unless QEMU crashes
877 out_state = "unexpected eof"
878 break
879 line = line + c
880 if c != "\n":
881 continue
882
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300883 # line contains a full line of data output from QEMU
Andrew Boie6acbe632015-07-17 12:03:52 -0700884 log_out_fp.write(line)
885 log_out_fp.flush()
886 line = line.strip()
887 verbose("QEMU: %s" % line)
888
Anas Nashif576be982017-12-23 20:20:27 -0500889 harness.handle(line)
890 if harness.state:
Anas Nashif39ae72b2018-08-29 01:45:38 -0400891 # if we have registered a fail make sure the state is not
892 # overridden by a false success message coming from the
893 # testsuite
894 if out_state != 'failed':
895 out_state = harness.state
896
897 # if we get some state, that means test is doing well, we reset
Marc Herbert9e573382019-07-03 12:49:42 -0700898 # the timeout and wait for 2 more seconds to catch anything
899 # printed late. We wait much longer if code
Andrew Boie095b82a2019-07-02 17:03:48 -0700900 # coverage is enabled since dumping this information can
901 # take some time.
Anas Nashiff29087e2019-01-25 09:37:38 -0500902 if not timeout_extended or harness.capture_coverage:
903 timeout_extended= True
904 if harness.capture_coverage:
Andrew Boied62e2292019-07-01 15:01:48 -0700905 timeout_time = time.time() + 30
Anas Nashiff29087e2019-01-25 09:37:38 -0500906 else:
Anas Nashif74dbe332019-01-17 17:11:37 -0500907 timeout_time = time.time() + 2
Andrew Boie6acbe632015-07-17 12:03:52 -0700908 line = ""
909
Anas Nashif83fc06a2019-06-22 11:04:10 -0400910 handler.record(harness)
911
912 handler_time = time.time() - start_time
Andrew Boie6acbe632015-07-17 12:03:52 -0700913 verbose("QEMU complete (%s) after %f seconds" %
Anas Nashif83fc06a2019-06-22 11:04:10 -0400914 (out_state, handler_time))
915 handler.set_state(out_state, handler_time)
Andrew Boie6acbe632015-07-17 12:03:52 -0700916
917 log_out_fp.close()
918 out_fp.close()
919 in_fp.close()
Anas Nashifd6476ee2019-04-11 11:40:09 -0400920 if os.path.exists(pid_fn):
921 pid = int(open(pid_fn).read())
922 os.unlink(pid_fn)
Andrew Boie6acbe632015-07-17 12:03:52 -0700923
Anas Nashifd6476ee2019-04-11 11:40:09 -0400924 try:
925 if pid:
926 os.kill(pid, signal.SIGTERM)
927 except ProcessLookupError:
928 # Oh well, as long as it's dead! User probably sent Ctrl-C
929 pass
Andrew Boie08ce5a52016-02-22 13:28:10 -0800930
Andrew Boie6acbe632015-07-17 12:03:52 -0700931 os.unlink(fifo_in)
932 os.unlink(fifo_out)
933
Anas Nashif83fc06a2019-06-22 11:04:10 -0400934 def handle(self):
Andrew Boie6acbe632015-07-17 12:03:52 -0700935 self.results = {}
Anas Nashif13773752018-07-06 18:20:23 -0500936 self.run = True
Andrew Boie6acbe632015-07-17 12:03:52 -0700937
938 # We pass this to QEMU which looks for fifos with .in and .out
939 # suffixes.
Anas Nashif83fc06a2019-06-22 11:04:10 -0400940 self.fifo_fn = os.path.join(self.instance.build_dir, "qemu-fifo")
Andrew Boie6acbe632015-07-17 12:03:52 -0700941
Anas Nashif83fc06a2019-06-22 11:04:10 -0400942 self.pid_fn = os.path.join(self.instance.build_dir, "qemu.pid")
Andrew Boie6acbe632015-07-17 12:03:52 -0700943 if os.path.exists(self.pid_fn):
944 os.unlink(self.pid_fn)
945
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500946 self.log_fn = self.log
Anas Nashif576be982017-12-23 20:20:27 -0500947
Anas Nashif83fc06a2019-06-22 11:04:10 -0400948 harness_import = HarnessImporter(self.instance.testcase.harness.capitalize())
Anas Nashif576be982017-12-23 20:20:27 -0500949 harness = harness_import.instance
Anas Nashifd3384fb2018-02-22 06:44:16 -0600950 harness.configure(self.instance)
951 self.thread = threading.Thread(name=self.name, target=QEMUHandler._thread,
Anas Nashif83fc06a2019-06-22 11:04:10 -0400952 args=(self, self.timeout, self.build_dir,
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300953 self.log_fn, self.fifo_fn,
Anas Nashif576be982017-12-23 20:20:27 -0500954 self.pid_fn, self.results, harness))
Anas Nashife0a6a0b2018-02-15 20:07:24 -0600955
956 self.instance.results = harness.tests
Andrew Boie6acbe632015-07-17 12:03:52 -0700957 self.thread.daemon = True
Anas Nashif83fc06a2019-06-22 11:04:10 -0400958 verbose("Spawning QEMUHandler Thread for %s" % self.name)
Andrew Boie6acbe632015-07-17 12:03:52 -0700959 self.thread.start()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400960 subprocess.call(["stty", "sane"])
Andrew Boie6acbe632015-07-17 12:03:52 -0700961
Andrew Boie6acbe632015-07-17 12:03:52 -0700962 def get_fifo(self):
963 return self.fifo_fn
964
Andrew Boie6acbe632015-07-17 12:03:52 -0700965class SizeCalculator:
Andrew Boie73b4ee62015-10-07 11:33:22 -0700966
Anas Nashif83fc06a2019-06-22 11:04:10 -0400967 alloc_sections = [
968 "bss",
969 "noinit",
970 "app_bss",
971 "app_noinit",
972 "ccm_bss",
973 "ccm_noinit"
974 ]
975
976 rw_sections = [
977 "datas",
978 "initlevel",
979 "exceptions",
980 "initshell",
981 "_static_thread_area",
982 "_k_timer_area",
983 "_k_mem_slab_area",
984 "_k_mem_pool_area",
985 "sw_isr_table",
986 "_k_sem_area",
987 "_k_mutex_area",
988 "app_shmem_regions",
989 "_k_fifo_area",
990 "_k_lifo_area",
991 "_k_stack_area",
992 "_k_msgq_area",
993 "_k_mbox_area",
994 "_k_pipe_area",
995 "net_if",
996 "net_if_dev",
997 "net_stack",
998 "net_l2_data",
999 "_k_queue_area",
1000 "_net_buf_pool_area",
1001 "app_datas",
1002 "kobject_data",
1003 "mmu_tables",
1004 "app_pad",
1005 "priv_stacks",
1006 "ccm_data",
1007 "usb_descriptor",
1008 "usb_data", "usb_bos_desc",
1009 'log_backends_sections',
1010 'log_dynamic_sections',
1011 'log_const_sections',
1012 "app_smem",
1013 'shell_root_cmds_sections',
1014 'log_const_sections',
1015 "font_entry_sections",
1016 "priv_stacks_noinit",
1017 "_TEXT_SECTION_NAME_2",
1018 "_GCOV_BSS_SECTION_NAME",
1019 "gcov",
1020 "nocache"
1021 ]
Anas Nashifdb9592a2018-10-08 10:19:41 -04001022
Andrew Boie73b4ee62015-10-07 11:33:22 -07001023 # These get copied into RAM only on non-XIP
Anas Nashif83fc06a2019-06-22 11:04:10 -04001024 ro_sections = [
1025 "text",
1026 "ctors",
1027 "init_array",
1028 "reset",
1029 "object_access",
1030 "rodata",
1031 "devconfig",
1032 "net_l2",
1033 "vector",
1034 "sw_isr_table",
1035 "_settings_handlers_area",
1036 "_bt_channels_area",
1037 "_bt_br_channels_area",
1038 "_bt_services_area",
1039 "vectors",
1040 "net_socket_register",
1041 "net_ppp_proto"
1042 ]
Andrew Boie73b4ee62015-10-07 11:33:22 -07001043
Andrew Boie52fef672016-11-29 12:21:59 -08001044 def __init__(self, filename, extra_sections):
Andrew Boie6acbe632015-07-17 12:03:52 -07001045 """Constructor
1046
Andrew Boiebbd670c2015-08-17 13:16:11 -07001047 @param filename Path to the output binary
1048 The <filename> is parsed by objdump to determine section sizes
Andrew Boie6acbe632015-07-17 12:03:52 -07001049 """
Andrew Boie6acbe632015-07-17 12:03:52 -07001050 # Make sure this is an ELF binary
Andrew Boiebbd670c2015-08-17 13:16:11 -07001051 with open(filename, "rb") as f:
Andrew Boie6acbe632015-07-17 12:03:52 -07001052 magic = f.read(4)
1053
Anas Nashifb4bdd662018-08-15 17:12:28 -05001054 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001055 if magic != b'\x7fELF':
Anas Nashifb4bdd662018-08-15 17:12:28 -05001056 raise SanityRuntimeError("%s is not an ELF binary" % filename)
1057 except Exception as e:
1058 print(str(e))
1059 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -07001060
1061 # Search for CONFIG_XIP in the ELF's list of symbols using NM and AWK.
Anas Nashif3ba1d432017-12-05 15:28:44 -05001062 # GREP can not be used as it returns an error if the symbol is not
1063 # found.
1064 is_xip_command = "nm " + filename + \
1065 " | awk '/CONFIG_XIP/ { print $3 }'"
1066 is_xip_output = subprocess.check_output(
1067 is_xip_command, shell=True, stderr=subprocess.STDOUT).decode(
1068 "utf-8").strip()
Anas Nashifb4bdd662018-08-15 17:12:28 -05001069 try:
1070 if is_xip_output.endswith("no symbols"):
1071 raise SanityRuntimeError("%s has no symbol information" % filename)
1072 except Exception as e:
1073 print(str(e))
1074 sys.exit(2)
1075
Andrew Boie6acbe632015-07-17 12:03:52 -07001076 self.is_xip = (len(is_xip_output) != 0)
1077
Andrew Boiebbd670c2015-08-17 13:16:11 -07001078 self.filename = filename
Andrew Boie73b4ee62015-10-07 11:33:22 -07001079 self.sections = []
1080 self.rom_size = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07001081 self.ram_size = 0
Andrew Boie52fef672016-11-29 12:21:59 -08001082 self.extra_sections = extra_sections
Andrew Boie6acbe632015-07-17 12:03:52 -07001083
1084 self._calculate_sizes()
1085
1086 def get_ram_size(self):
1087 """Get the amount of RAM the application will use up on the device
1088
1089 @return amount of RAM, in bytes
1090 """
Andrew Boie73b4ee62015-10-07 11:33:22 -07001091 return self.ram_size
Andrew Boie6acbe632015-07-17 12:03:52 -07001092
1093 def get_rom_size(self):
1094 """Get the size of the data that this application uses on device's flash
1095
1096 @return amount of ROM, in bytes
1097 """
Andrew Boie73b4ee62015-10-07 11:33:22 -07001098 return self.rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -07001099
1100 def unrecognized_sections(self):
1101 """Get a list of sections inside the binary that weren't recognized
1102
David B. Kinder29963c32017-06-16 12:32:42 -07001103 @return list of unrecognized section names
Andrew Boie6acbe632015-07-17 12:03:52 -07001104 """
1105 slist = []
Andrew Boie73b4ee62015-10-07 11:33:22 -07001106 for v in self.sections:
Andrew Boie6acbe632015-07-17 12:03:52 -07001107 if not v["recognized"]:
Andrew Boie73b4ee62015-10-07 11:33:22 -07001108 slist.append(v["name"])
Andrew Boie6acbe632015-07-17 12:03:52 -07001109 return slist
1110
1111 def _calculate_sizes(self):
1112 """ Calculate RAM and ROM usage by section """
Andrew Boiebbd670c2015-08-17 13:16:11 -07001113 objdump_command = "objdump -h " + self.filename
Anas Nashif3ba1d432017-12-05 15:28:44 -05001114 objdump_output = subprocess.check_output(
1115 objdump_command, shell=True).decode("utf-8").splitlines()
Andrew Boie6acbe632015-07-17 12:03:52 -07001116
1117 for line in objdump_output:
1118 words = line.split()
1119
Anas Nashif83fc06a2019-06-22 11:04:10 -04001120 if not words: # Skip lines that are too short
Andrew Boie6acbe632015-07-17 12:03:52 -07001121 continue
1122
1123 index = words[0]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001124 if not index[0].isdigit(): # Skip lines that do not start
Andrew Boie6acbe632015-07-17 12:03:52 -07001125 continue # with a digit
1126
1127 name = words[1] # Skip lines with section names
Anas Nashif83fc06a2019-06-22 11:04:10 -04001128 if name[0] == '.': # starting with '.'
Andrew Boie6acbe632015-07-17 12:03:52 -07001129 continue
1130
Andrew Boie73b4ee62015-10-07 11:33:22 -07001131 # TODO this doesn't actually reflect the size in flash or RAM as
1132 # it doesn't include linker-imposed padding between sections.
1133 # It is close though.
Andrew Boie6acbe632015-07-17 12:03:52 -07001134 size = int(words[2], 16)
Andrew Boie9882dcd2015-10-07 14:25:51 -07001135 if size == 0:
1136 continue
1137
Andrew Boie73b4ee62015-10-07 11:33:22 -07001138 load_addr = int(words[4], 16)
1139 virt_addr = int(words[3], 16)
Andrew Boie6acbe632015-07-17 12:03:52 -07001140
1141 # Add section to memory use totals (for both non-XIP and XIP scenarios)
Andrew Boie6acbe632015-07-17 12:03:52 -07001142 # Unrecognized section names are not included in the calculations.
Andrew Boie6acbe632015-07-17 12:03:52 -07001143 recognized = True
Andrew Boie73b4ee62015-10-07 11:33:22 -07001144 if name in SizeCalculator.alloc_sections:
1145 self.ram_size += size
1146 stype = "alloc"
1147 elif name in SizeCalculator.rw_sections:
1148 self.ram_size += size
1149 self.rom_size += size
1150 stype = "rw"
1151 elif name in SizeCalculator.ro_sections:
1152 self.rom_size += size
1153 if not self.is_xip:
1154 self.ram_size += size
1155 stype = "ro"
Andrew Boie6acbe632015-07-17 12:03:52 -07001156 else:
Andrew Boie73b4ee62015-10-07 11:33:22 -07001157 stype = "unknown"
Andrew Boie52fef672016-11-29 12:21:59 -08001158 if name not in self.extra_sections:
1159 recognized = False
Andrew Boie6acbe632015-07-17 12:03:52 -07001160
Anas Nashif3ba1d432017-12-05 15:28:44 -05001161 self.sections.append({"name": name, "load_addr": load_addr,
1162 "size": size, "virt_addr": virt_addr,
1163 "type": stype, "recognized": recognized})
Andrew Boie6acbe632015-07-17 12:03:52 -07001164
1165
Andrew Boie6acbe632015-07-17 12:03:52 -07001166# "list" - List of strings
1167# "list:<type>" - List of <type>
1168# "set" - Set of unordered, unique strings
1169# "set:<type>" - Set of <type>
1170# "float" - Floating point
1171# "int" - Integer
1172# "bool" - Boolean
1173# "str" - String
1174
1175# XXX Be sure to update __doc__ if you change any of this!!
1176
Anas Nashif83fc06a2019-06-22 11:04:10 -04001177platform_valid_keys = {
Anas Nashif924a4e72018-10-18 12:25:55 -04001178 "supported_toolchains": {"type": "list", "default": []},
1179 "env": {"type": "list", "default": []}
1180 }
Andrew Boie6acbe632015-07-17 12:03:52 -07001181
Anas Nashif3ba1d432017-12-05 15:28:44 -05001182testcase_valid_keys = {"tags": {"type": "set", "required": False},
1183 "type": {"type": "str", "default": "integration"},
1184 "extra_args": {"type": "list"},
1185 "extra_configs": {"type": "list"},
1186 "build_only": {"type": "bool", "default": False},
1187 "build_on_all": {"type": "bool", "default": False},
1188 "skip": {"type": "bool", "default": False},
1189 "slow": {"type": "bool", "default": False},
1190 "timeout": {"type": "int", "default": 60},
1191 "min_ram": {"type": "int", "default": 8},
1192 "depends_on": {"type": "set"},
1193 "min_flash": {"type": "int", "default": 32},
1194 "arch_whitelist": {"type": "set"},
1195 "arch_exclude": {"type": "set"},
1196 "extra_sections": {"type": "list", "default": []},
1197 "platform_exclude": {"type": "set"},
1198 "platform_whitelist": {"type": "set"},
1199 "toolchain_exclude": {"type": "set"},
1200 "toolchain_whitelist": {"type": "set"},
Anas Nashifab940162017-12-08 10:17:57 -05001201 "filter": {"type": "str"},
Anas Nashif576be982017-12-23 20:20:27 -05001202 "harness": {"type": "str"},
Praful Swarnakarcf89e282018-09-13 02:58:28 +05301203 "harness_config": {"type": "map", "default": {}}
Anas Nashifab940162017-12-08 10:17:57 -05001204 }
Andrew Boie6acbe632015-07-17 12:03:52 -07001205
Andrew Boie6acbe632015-07-17 12:03:52 -07001206class SanityConfigParser:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001207 """Class to read test case files with semantic checking
Andrew Boie6acbe632015-07-17 12:03:52 -07001208 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001209
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001210 def __init__(self, filename, schema):
Andrew Boie6acbe632015-07-17 12:03:52 -07001211 """Instantiate a new SanityConfigParser object
1212
Anas Nashifa792a3d2017-04-04 18:47:49 -04001213 @param filename Source .yaml file to read
Andrew Boie6acbe632015-07-17 12:03:52 -07001214 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001215 self.data = {}
1216 self.schema = schema
Andrew Boie6acbe632015-07-17 12:03:52 -07001217 self.filename = filename
Anas Nashif255625b2017-12-05 15:08:26 -05001218 self.tests = {}
1219 self.common = {}
Anas Nashif83fc06a2019-06-22 11:04:10 -04001220
1221 def load(self):
1222 self.data = scl.yaml_load_verify(self.filename, self.schema)
1223
Anas Nashif255625b2017-12-05 15:08:26 -05001224 if 'tests' in self.data:
1225 self.tests = self.data['tests']
1226 if 'common' in self.data:
1227 self.common = self.data['common']
Andrew Boie6acbe632015-07-17 12:03:52 -07001228
Anas Nashif83fc06a2019-06-22 11:04:10 -04001229
Andrew Boie6acbe632015-07-17 12:03:52 -07001230 def _cast_value(self, value, typestr):
Anas Nashif3ba1d432017-12-05 15:28:44 -05001231 if isinstance(value, str):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001232 v = value.strip()
Andrew Boie6acbe632015-07-17 12:03:52 -07001233 if typestr == "str":
1234 return v
1235
1236 elif typestr == "float":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001237 return float(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001238
1239 elif typestr == "int":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001240 return int(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001241
1242 elif typestr == "bool":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001243 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001244
Anas Nashif3ba1d432017-12-05 15:28:44 -05001245 elif typestr.startswith("list") and isinstance(value, list):
Anas Nashiffa695d22017-10-04 16:14:27 -04001246 return value
Anas Nashif3ba1d432017-12-05 15:28:44 -05001247 elif typestr.startswith("list") and isinstance(value, str):
Andrew Boie6acbe632015-07-17 12:03:52 -07001248 vs = v.split()
1249 if len(typestr) > 4 and typestr[4] == ":":
1250 return [self._cast_value(vsi, typestr[5:]) for vsi in vs]
1251 else:
1252 return vs
1253
1254 elif typestr.startswith("set"):
1255 vs = v.split()
1256 if len(typestr) > 3 and typestr[3] == ":":
Anas Nashif83fc06a2019-06-22 11:04:10 -04001257 return {self._cast_value(vsi, typestr[4:]) for vsi in vs}
Andrew Boie6acbe632015-07-17 12:03:52 -07001258 else:
1259 return set(vs)
1260
Anas Nashif576be982017-12-23 20:20:27 -05001261 elif typestr.startswith("map"):
1262 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001263 else:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001264 raise ConfigurationError(
1265 self.filename, "unknown type '%s'" % value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001266
Anas Nashifb4754ed2017-12-05 17:27:58 -05001267 def get_test(self, name, valid_keys):
1268 """Get a dictionary representing the keys/values within a test
Andrew Boie6acbe632015-07-17 12:03:52 -07001269
Anas Nashifb4754ed2017-12-05 17:27:58 -05001270 @param name The test in the .yaml file to retrieve data from
Andrew Boie6acbe632015-07-17 12:03:52 -07001271 @param valid_keys A dictionary representing the intended semantics
Anas Nashifb4754ed2017-12-05 17:27:58 -05001272 for this test. Each key in this dictionary is a key that could
Anas Nashifa792a3d2017-04-04 18:47:49 -04001273 be specified, if a key is given in the .yaml file which isn't in
Andrew Boie6acbe632015-07-17 12:03:52 -07001274 here, it will generate an error. Each value in this dictionary
1275 is another dictionary containing metadata:
1276
1277 "default" - Default value if not given
1278 "type" - Data type to convert the text value to. Simple types
1279 supported are "str", "float", "int", "bool" which will get
1280 converted to respective Python data types. "set" and "list"
1281 may also be specified which will split the value by
1282 whitespace (but keep the elements as strings). finally,
1283 "list:<type>" and "set:<type>" may be given which will
1284 perform a type conversion after splitting the value up.
1285 "required" - If true, raise an error if not defined. If false
1286 and "default" isn't specified, a type conversion will be
1287 done on an empty string
Anas Nashifb4754ed2017-12-05 17:27:58 -05001288 @return A dictionary containing the test key-value pairs with
Andrew Boie6acbe632015-07-17 12:03:52 -07001289 type conversion and default values filled in per valid_keys
1290 """
1291
1292 d = {}
Anas Nashif255625b2017-12-05 15:08:26 -05001293 for k, v in self.common.items():
Anas Nashiffa695d22017-10-04 16:14:27 -04001294 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001295
Anas Nashifb4754ed2017-12-05 17:27:58 -05001296 for k, v in self.tests[name].items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001297 if k not in valid_keys:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001298 raise ConfigurationError(
1299 self.filename,
1300 "Unknown config key '%s' in definition for '%s'" %
Anas Nashifb4754ed2017-12-05 17:27:58 -05001301 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001302
Anas Nashiffa695d22017-10-04 16:14:27 -04001303 if k in d:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001304 if isinstance(d[k], str):
Paul Sokolovsky8c6d2e22019-08-29 13:20:57 +03001305 # By default, we just concatenate string values of keys
1306 # which appear both in "common" and per-test sections,
1307 # but some keys are handled in adhoc way based on their
1308 # semantics.
1309 if k == "filter":
1310 d[k] = "(%s) and (%s)" % (d[k], v)
1311 else:
1312 d[k] += " " + v
Anas Nashiffa695d22017-10-04 16:14:27 -04001313 else:
1314 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001315
Andrew Boie08ce5a52016-02-22 13:28:10 -08001316 for k, kinfo in valid_keys.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001317 if k not in d:
1318 if "required" in kinfo:
1319 required = kinfo["required"]
1320 else:
1321 required = False
1322
1323 if required:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001324 raise ConfigurationError(
1325 self.filename,
Anas Nashifb4754ed2017-12-05 17:27:58 -05001326 "missing required value for '%s' in test '%s'" %
1327 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001328 else:
1329 if "default" in kinfo:
1330 default = kinfo["default"]
1331 else:
1332 default = self._cast_value("", kinfo["type"])
1333 d[k] = default
1334 else:
1335 try:
1336 d[k] = self._cast_value(d[k], kinfo["type"])
Anas Nashif83fc06a2019-06-22 11:04:10 -04001337 except ValueError:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001338 raise ConfigurationError(
Anas Nashifb4754ed2017-12-05 17:27:58 -05001339 self.filename, "bad %s value '%s' for key '%s' in name '%s'" %
1340 (kinfo["type"], d[k], k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001341
1342 return d
1343
1344
1345class Platform:
1346 """Class representing metadata for a particular platform
1347
Anas Nashifc7406082015-12-13 15:00:31 -05001348 Maps directly to BOARD when building"""
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001349
Anas Nashif83fc06a2019-06-22 11:04:10 -04001350 platform_schema = scl.yaml_load(os.path.join(ZEPHYR_BASE,
1351 "scripts","sanity_chk","platform-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001352
Anas Nashif83fc06a2019-06-22 11:04:10 -04001353 def __init__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001354 """Constructor.
1355
Andrew Boie6acbe632015-07-17 12:03:52 -07001356 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001357
1358 self.name = ""
1359 self.sanitycheck = True
1360 # if no RAM size is specified by the board, take a default of 128K
1361 self.ram = 128
1362
1363 self.ignore_tags = []
1364 self.default = False
1365 # if no flash size is specified by the board, take a default of 512K
1366 self.flash = 512
1367 self.supported = set()
1368
1369 self.arch = ""
1370 self.type = "na"
1371 self.simulation = "na"
1372 self.supported_toolchains = []
1373 self.env = []
1374 self.env_satisfied = True
1375 self.filter_data = dict()
1376
1377 def load(self, platform_file):
1378 scp = SanityConfigParser(platform_file, self.platform_schema)
1379 scp.load()
Anas Nashif255625b2017-12-05 15:08:26 -05001380 data = scp.data
Anas Nashifa792a3d2017-04-04 18:47:49 -04001381
Anas Nashif255625b2017-12-05 15:08:26 -05001382 self.name = data['identifier']
Anas Nashiff3d48e12018-07-24 08:14:42 -05001383 self.sanitycheck = data.get("sanitycheck", True)
Anas Nashifa792a3d2017-04-04 18:47:49 -04001384 # if no RAM size is specified by the board, take a default of 128K
Anas Nashif255625b2017-12-05 15:08:26 -05001385 self.ram = data.get("ram", 128)
1386 testing = data.get("testing", {})
Anas Nashifa792a3d2017-04-04 18:47:49 -04001387 self.ignore_tags = testing.get("ignore_tags", [])
1388 self.default = testing.get("default", False)
1389 # if no flash size is specified by the board, take a default of 512K
Anas Nashif255625b2017-12-05 15:08:26 -05001390 self.flash = data.get("flash", 512)
Anas Nashif95276932017-07-31 08:47:41 -04001391 self.supported = set()
Anas Nashif255625b2017-12-05 15:08:26 -05001392 for supp_feature in data.get("supported", []):
Anas Nashif95276932017-07-31 08:47:41 -04001393 for item in supp_feature.split(":"):
1394 self.supported.add(item)
1395
Anas Nashif255625b2017-12-05 15:08:26 -05001396 self.arch = data['arch']
Anas Nashifcc164222017-12-26 11:02:46 -05001397 self.type = data.get('type', "na")
Anas Nashif8acdbd72018-01-04 14:15:22 -05001398 self.simulation = data.get('simulation', "na")
Anas Nashif255625b2017-12-05 15:08:26 -05001399 self.supported_toolchains = data.get("toolchain", [])
Anas Nashif924a4e72018-10-18 12:25:55 -04001400 self.env = data.get("env", [])
1401 self.env_satisfied = True
1402 for env in self.env:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001403 if not os.environ.get(env, None):
Anas Nashif924a4e72018-10-18 12:25:55 -04001404 self.env_satisfied = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001405
Andrew Boie6acbe632015-07-17 12:03:52 -07001406
Andrew Boie6acbe632015-07-17 12:03:52 -07001407 def __repr__(self):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001408 return "<%s on %s>" % (self.name, self.arch)
Andrew Boie6acbe632015-07-17 12:03:52 -07001409
Anas Nashif83fc06a2019-06-22 11:04:10 -04001410class TestCase(object):
Andrew Boie6acbe632015-07-17 12:03:52 -07001411 """Class representing a test application
1412 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001413
Anas Nashif83fc06a2019-06-22 11:04:10 -04001414
1415 def __init__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001416 """TestCase constructor.
1417
Anas Nashif877d3ca2017-12-05 17:39:29 -05001418 This gets called by TestSuite as it finds and reads test yaml files.
Anas Nashifa792a3d2017-04-04 18:47:49 -04001419 Multiple TestCase instances may be generated from a single testcase.yaml,
Anas Nashif877d3ca2017-12-05 17:39:29 -05001420 each one corresponds to an entry within that file.
Andrew Boie6acbe632015-07-17 12:03:52 -07001421
Andrew Boie6acbe632015-07-17 12:03:52 -07001422 We need to have a unique name for every single test case. Since
Anas Nashifa792a3d2017-04-04 18:47:49 -04001423 a testcase.yaml can define multiple tests, the canonical name for
Andrew Boie6acbe632015-07-17 12:03:52 -07001424 the test case is <workdir>/<name>.
1425
Marc Herbert1c8632c2019-04-15 17:58:45 -07001426 @param testcase_root os.path.abspath() of one of the --testcase-root
1427 @param workdir Sub-directory of testcase_root where the
1428 .yaml test configuration file was found
Anas Nashif877d3ca2017-12-05 17:39:29 -05001429 @param name Name of this test case, corresponding to the entry name
Andrew Boie6acbe632015-07-17 12:03:52 -07001430 in the test case configuration file. For many test cases that just
1431 define one test, can be anything and is usually "test". This is
1432 really only used to distinguish between different cases when
Anas Nashifa792a3d2017-04-04 18:47:49 -04001433 the testcase.yaml defines multiple tests
Anas Nashif877d3ca2017-12-05 17:39:29 -05001434 @param tc_dict Dictionary with test values for this test case
Anas Nashifa792a3d2017-04-04 18:47:49 -04001435 from the testcase.yaml file
Andrew Boie6acbe632015-07-17 12:03:52 -07001436 """
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001437
Anas Nashif83fc06a2019-06-22 11:04:10 -04001438 self.id = ""
1439 self.source_dir = ""
1440 self.yamlfile = ""
Anas Nashifaae71d72018-04-21 22:26:48 -05001441 self.cases = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04001442 self.name = ""
Anas Nashifbd166f42017-09-02 12:32:08 -04001443
Anas Nashif83fc06a2019-06-22 11:04:10 -04001444 self.type = None
1445 self.tags = None
1446 self.extra_args = None
1447 self.extra_configs = None
1448 self.arch_whitelist = None
1449 self.arch_exclude = None
1450 self.skip = None
1451 self.platform_exclude = None
1452 self.platform_whitelist = None
1453 self.toolchain_exclude = None
1454 self.toolchain_whitelist = None
1455 self.tc_filter = None
1456 self.timeout = 60
1457 self.harness = ""
1458 self.harness_config = {}
1459 self.build_only = True
1460 self.build_on_all = False
1461 self.slow = False
1462 self.min_ram = None
1463 self.depends_on = None
1464 self.min_flash = None
1465 self.extra_sections = None
Andrew Boie6acbe632015-07-17 12:03:52 -07001466
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001467
Anas Nashif83fc06a2019-06-22 11:04:10 -04001468 @staticmethod
1469 def get_unique(testcase_root, workdir, name):
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001470
Marc Herbert1c8632c2019-04-15 17:58:45 -07001471 canonical_testcase_root = os.path.realpath(testcase_root)
Tyler Hallbb359252019-06-19 02:11:35 -04001472 if Path(canonical_zephyr_base) in Path(canonical_testcase_root).parents:
Marc Herbert1c8632c2019-04-15 17:58:45 -07001473 # This is in ZEPHYR_BASE, so include path in name for uniqueness
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001474 # FIXME: We should not depend on path of test for unique names.
Marc Herbert1c8632c2019-04-15 17:58:45 -07001475 relative_tc_root = os.path.relpath(canonical_testcase_root,
1476 start=canonical_zephyr_base)
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001477 else:
Marc Herbert1c8632c2019-04-15 17:58:45 -07001478 relative_tc_root = ""
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001479
Marc Herbert1c8632c2019-04-15 17:58:45 -07001480 # workdir can be "."
1481 unique = os.path.normpath(os.path.join(relative_tc_root, workdir, name))
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001482 return unique
1483
Anas Nashif83fc06a2019-06-22 11:04:10 -04001484 @staticmethod
1485 def scan_file(inf_name):
Anas Nashifaae71d72018-04-21 22:26:48 -05001486 suite_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001487 # do not match until end-of-line, otherwise we won't allow
1488 # stc_regex below to catch the ones that are declared in the same
1489 # line--as we only search starting the end of this match
1490 br"^\s*ztest_test_suite\(\s*(?P<suite_name>[a-zA-Z0-9_]+)\s*,",
Anas Nashifaae71d72018-04-21 22:26:48 -05001491 re.MULTILINE)
1492 stc_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001493 br"^\s*" # empy space at the beginning is ok
1494 # catch the case where it is declared in the same sentence, e.g:
1495 #
1496 # ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
1497 br"(?:ztest_test_suite\([a-zA-Z0-9_]+,\s*)?"
1498 # Catch ztest[_user]_unit_test-[_setup_teardown](TESTNAME)
1499 br"ztest_(?:user_)?unit_test(?:_setup_teardown)?"
1500 # Consume the argument that becomes the extra testcse
1501 br"\(\s*"
1502 br"(?P<stc_name>[a-zA-Z0-9_]+)"
1503 # _setup_teardown() variant has two extra arguments that we ignore
1504 br"(?:\s*,\s*[a-zA-Z0-9_]+\s*,\s*[a-zA-Z0-9_]+)?"
1505 br"\s*\)",
1506 # We don't check how it finishes; we don't care
Anas Nashifaae71d72018-04-21 22:26:48 -05001507 re.MULTILINE)
1508 suite_run_regex = re.compile(
1509 br"^\s*ztest_run_test_suite\((?P<suite_name>[a-zA-Z0-9_]+)\)",
1510 re.MULTILINE)
1511 achtung_regex = re.compile(
1512 br"(#ifdef|#endif)",
1513 re.MULTILINE)
1514 warnings = None
1515
1516 with open(inf_name) as inf:
1517 with contextlib.closing(mmap.mmap(inf.fileno(), 0, mmap.MAP_PRIVATE,
1518 mmap.PROT_READ, 0)) as main_c:
Anas Nashifaae71d72018-04-21 22:26:48 -05001519 suite_regex_match = suite_regex.search(main_c)
1520 if not suite_regex_match:
1521 # can't find ztest_test_suite, maybe a client, because
1522 # it includes ztest.h
1523 return None, None
1524
1525 suite_run_match = suite_run_regex.search(main_c)
1526 if not suite_run_match:
1527 raise ValueError("can't find ztest_run_test_suite")
1528
1529 achtung_matches = re.findall(
1530 achtung_regex,
1531 main_c[suite_regex_match.end():suite_run_match.start()])
1532 if achtung_matches:
1533 warnings = "found invalid %s in ztest_test_suite()" \
Anas Nashif83fc06a2019-06-22 11:04:10 -04001534 % ", ".join({match.decode() for match in achtung_matches})
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001535 _matches = re.findall(
Anas Nashifaae71d72018-04-21 22:26:48 -05001536 stc_regex,
1537 main_c[suite_regex_match.end():suite_run_match.start()])
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001538 matches = [ match.decode().replace("test_", "") for match in _matches ]
Anas Nashifaae71d72018-04-21 22:26:48 -05001539 return matches, warnings
1540
1541 def scan_path(self, path):
1542 subcases = []
1543 for filename in glob.glob(os.path.join(path, "src", "*.c")):
1544 try:
1545 _subcases, warnings = self.scan_file(filename)
1546 if warnings:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001547 error("%s: %s" % (filename, warnings))
Anas Nashifaae71d72018-04-21 22:26:48 -05001548 if _subcases:
1549 subcases += _subcases
1550 except ValueError as e:
Flavio Ceolinbf878ce2019-04-19 22:24:09 -07001551 error("%s: can't find: %s" % (filename, e))
Anas Nashifaae71d72018-04-21 22:26:48 -05001552 return subcases
1553
Anas Nashif83fc06a2019-06-22 11:04:10 -04001554 def parse_subcases(self, test_path):
1555 results = self.scan_path(os.path.dirname(test_path))
Anas Nashifaae71d72018-04-21 22:26:48 -05001556 for sub in results:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001557 name = "{}.{}".format(self.id, sub)
Anas Nashifaae71d72018-04-21 22:26:48 -05001558 self.cases.append(name)
1559
Anas Nashiff16e92c2019-03-31 16:58:12 -04001560 if not results:
1561 self.cases.append(self.id)
1562
Anas Nashifaae71d72018-04-21 22:26:48 -05001563
Anas Nashif75547e22018-02-24 08:32:14 -06001564 def __str__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001565 return self.name
1566
1567
Andrew Boie6acbe632015-07-17 12:03:52 -07001568class TestInstance:
1569 """Class representing the execution of a particular TestCase on a platform
1570
1571 @param test The TestCase object we want to build/execute
1572 @param platform Platform object that we want to build and run against
1573 @param base_outdir Base directory for all test results. The actual
1574 out directory used is <outdir>/<platform>/<test case name>
1575 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001576
Anas Nashif83fc06a2019-06-22 11:04:10 -04001577 def __init__(self, testcase, platform, base_outdir):
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001578
Anas Nashif83fc06a2019-06-22 11:04:10 -04001579 self.testcase = testcase
1580 self.platform = platform
1581
1582 self.status = None
1583 self.reason = "N/A"
1584 self.metrics = dict()
1585 self.handler = None
1586
1587
1588 self.name = os.path.join(platform.name, testcase.name)
1589 self.build_dir = os.path.join(base_outdir, platform.name, testcase.name)
1590
1591 self.build_only = self.check_build_or_run()
1592 self.run = not self.build_only
1593
Anas Nashife0a6a0b2018-02-15 20:07:24 -06001594 self.results = {}
Andrew Boie6acbe632015-07-17 12:03:52 -07001595
Marc Herbert0f7255c2019-04-05 14:14:21 -07001596 def __lt__(self, other):
1597 return self.name < other.name
1598
Anas Nashif83fc06a2019-06-22 11:04:10 -04001599 def check_build_or_run(self):
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001600
Anas Nashif83fc06a2019-06-22 11:04:10 -04001601 build_only = True
1602
1603 # we asked for build-only on the command line
1604 if options.build_only:
1605 return True
1606
1607 # The testcase is designed to be build only.
1608 if self.testcase.build_only:
1609 return True
1610
1611 # Do not run slow tests:
1612 skip_slow = self.testcase.slow and not options.enable_slow
1613 if skip_slow:
1614 return True
1615
1616 runnable =bool(self.testcase.type == "unit" or \
1617 self.platform.type == "native" or \
1618 self.platform.simulation in ["nsim", "renode", "qemu"] or \
1619 options.device_testing)
1620
1621 if self.platform.simulation == "nsim":
1622 if not find_executable("nsimdrv"):
1623 runnable = False
1624
1625 if self.platform.simulation == "renode":
1626 if not find_executable("renode"):
1627 runnable = False
1628
1629 # console harness allows us to run the test and capture data.
1630 if self.testcase.harness == 'console':
1631
1632 # if we have a fixture that is also being supplied on the
1633 # command-line, then we need to run the test, not just build it.
1634 if "fixture" in self.testcase.harness_config:
1635 fixture = self.testcase.harness_config['fixture']
1636 if fixture in options.fixture:
1637 build_only = False
1638 else:
1639 build_only = True
1640 else:
1641 build_only = False
1642 elif self.testcase.harness:
1643 build_only = True
1644 else:
1645 build_only = False
1646
1647 return not (not build_only and runnable)
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001648
Anas Nashifdbd76492018-11-23 20:24:19 -05001649 def create_overlay(self, platform):
Marc Herbertc7633de2019-07-06 15:52:31 -07001650 # Create this in a "sanitycheck/" subdirectory otherwise this
1651 # will pass this overlay to kconfig.py *twice* and kconfig.cmake
1652 # will silently give that second time precedence over any
1653 # --extra-args=CONFIG_*
Anas Nashif83fc06a2019-06-22 11:04:10 -04001654 subdir = os.path.join(self.build_dir, "sanitycheck")
Marc Herbertc7633de2019-07-06 15:52:31 -07001655 os.makedirs(subdir, exist_ok=True)
1656 file = os.path.join(subdir, "testcase_extra.conf")
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001657 with open(file, "w") as f:
1658 content = ""
Anas Nashif3cbffef2018-11-07 23:50:54 -05001659
Anas Nashif83fc06a2019-06-22 11:04:10 -04001660 if self.testcase.extra_configs:
1661 content = "\n".join(self.testcase.extra_configs)
Anas Nashif3cbffef2018-11-07 23:50:54 -05001662
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001663 if options.enable_coverage:
1664 if platform in options.coverage_platform:
1665 content = content + "\nCONFIG_COVERAGE=y"
1666
1667 f.write(content)
Anas Nashiffa695d22017-10-04 16:14:27 -04001668
Andrew Boie6acbe632015-07-17 12:03:52 -07001669 def calculate_sizes(self):
1670 """Get the RAM/ROM sizes of a test case.
1671
1672 This can only be run after the instance has been executed by
1673 MakeGenerator, otherwise there won't be any binaries to measure.
1674
1675 @return A SizeCalculator object
1676 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001677 fns = glob.glob(os.path.join(self.build_dir, "zephyr", "*.elf"))
1678 fns.extend(glob.glob(os.path.join(self.build_dir, "zephyr", "*.exe")))
Anas Nashiff8dcad42016-10-27 18:10:08 -04001679 fns = [x for x in fns if not x.endswith('_prebuilt.elf')]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001680 if len(fns) != 1:
Andrew Boie5d4eb782015-10-02 10:04:56 -07001681 raise BuildError("Missing/multiple output ELF binary")
Anas Nashif83fc06a2019-06-22 11:04:10 -04001682
1683 return SizeCalculator(fns[0], self.testcase.extra_sections)
Andrew Boie6acbe632015-07-17 12:03:52 -07001684
1685 def __repr__(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -04001686 return "<TestCase %s on %s>" % (self.testcase.name, self.platform.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07001687
1688
Anas Nashif83fc06a2019-06-22 11:04:10 -04001689class CMake():
Andrew Boie4ef16c52015-08-28 12:36:03 -07001690
Anas Nashif83fc06a2019-06-22 11:04:10 -04001691 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
1692 dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
1693
1694 def __init__(self, testcase, platform, source_dir, build_dir):
1695
1696 self.cwd = None
1697 self.capture_output = True
1698
1699 self.defconfig = {}
1700 self.cmake_cache = {}
1701 self.devicetree = {}
1702
1703 self.instance = None
1704 self.testcase = testcase
1705 self.platform = platform
1706 self.source_dir = source_dir
1707 self.build_dir = build_dir
1708 self.log = "build.log"
1709
1710 def parse_generated(self):
1711 self.defconfig = {}
1712 return {}
1713
1714 def run_build(self, args=[]):
1715
1716 verbose("Building %s for %s" % (self.source_dir, self.platform.name))
1717
1718 cmake_args = []
1719 cmake_args.extend(args)
1720 cmake = shutil.which('cmake')
1721 cmd = [cmake] + cmake_args
1722 kwargs = dict()
1723
1724 if self.capture_output:
1725 kwargs['stdout'] = subprocess.PIPE
1726 # CMake sends the output of message() to stderr unless it's STATUS
1727 kwargs['stderr'] = subprocess.STDOUT
1728
1729 if self.cwd:
1730 kwargs['cwd'] = self.cwd
1731
1732 p = subprocess.Popen(cmd, **kwargs)
1733 out, _ = p.communicate()
1734
1735 results = {}
1736 if p.returncode == 0:
1737 msg = "Finished building %s for %s" %(self.source_dir, self.platform.name)
1738
1739 self.instance.status = "passed"
1740 self.instance.reason = ""
1741 results = {'msg': msg, "returncode": p.returncode, "instance": self.instance}
1742
1743 if out:
1744 log_msg = out.decode(sys.getdefaultencoding())
1745 with open(os.path.join(self.build_dir, self.log), "a") as log:
1746 log.write(log_msg)
1747
1748 else:
1749 return None
1750 else:
1751 # A real error occurred, raise an exception
1752 if out:
1753 log_msg = out.decode(sys.getdefaultencoding())
1754 with open(os.path.join(self.build_dir, self.log), "a") as log:
1755 log.write(log_msg)
1756
1757 overflow_flash = "region `FLASH' overflowed by"
1758 overflow_ram = "region `RAM' overflowed by"
1759
1760 if log_msg:
1761 if log_msg.find(overflow_flash) > 0 or log_msg.find(overflow_ram) > 0:
1762 verbose("RAM/ROM Overflow")
1763 self.instance.status = "skipped"
1764 self.instance.reason = "overflow"
1765 else:
1766 self.instance.status = "failed"
1767 self.instance.reason = "Build failure"
1768
1769 results = {
1770 "returncode": p.returncode,
1771 "instance": self.instance,
1772 }
1773
1774 return results
1775
1776 def run_cmake(self, args=[]):
1777
1778 verbose("Running cmake on %s for %s" %(self.source_dir, self.platform.name))
1779
Anas Nashifa5984ab2019-10-22 07:36:24 -07001780 ldflags="-Wl,--fatal-warnings"
1781
1782 #fixme: add additional cflags based on options
1783 cmake_args = [
1784 '-B{}'.format(self.build_dir),
1785 '-S{}'.format(self.source_dir),
1786 '-DEXTRA_CFLAGS="-Werror ',
1787 '-DEXTRA_AFLAGS=-Wa,--fatal-warnings',
1788 '-DEXTRA_LDFLAGS="{}'.format(ldflags),
1789 '-G{}'.format(get_generator()[1])
1790 ]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001791
1792 args = ["-D{}".format(a.replace('"', '')) for a in args]
1793 cmake_args.extend(args)
1794
1795 cmake_opts = ['-DBOARD={}'.format(self.platform.name)]
1796 cmake_args.extend(cmake_opts)
1797
1798 cmake = shutil.which('cmake')
1799 cmd = [cmake] + cmake_args
1800 kwargs = dict()
1801
1802 if self.capture_output:
1803 kwargs['stdout'] = subprocess.PIPE
1804 # CMake sends the output of message() to stderr unless it's STATUS
1805 kwargs['stderr'] = subprocess.STDOUT
1806
1807 if self.cwd:
1808 kwargs['cwd'] = self.cwd
1809
1810 p = subprocess.Popen(cmd, **kwargs)
1811 out, _ = p.communicate()
1812
1813 if p.returncode == 0:
1814 filter_results = self.parse_generated()
1815 msg = "Finished building %s for %s" %(self.source_dir, self.platform.name)
1816
1817 results = {'msg': msg, 'filter': filter_results}
1818
1819 else:
1820 self.instance.status = "failed"
1821 self.instance.reason = "Cmake build failure"
1822 results = {"returncode": p.returncode}
1823
1824
1825 if out:
1826 with open(os.path.join(self.build_dir, self.log), "a") as log:
1827 log_msg = out.decode(sys.getdefaultencoding())
1828 log.write(log_msg)
1829
1830 return results
1831
1832
1833class FilterBuilder(CMake):
1834
1835 def __init__(self, testcase, platform, source_dir, build_dir):
1836 super().__init__(testcase, platform, source_dir, build_dir)
1837
1838 self.log = "config-sanitycheck.log"
1839
1840 def parse_generated(self):
1841
1842 if self.platform.name == "unit_testing":
1843 return {}
1844
1845 _generated_dt_confg = "include/generated/generated_dts_board.conf"
1846
1847 cmake_cache_path = os.path.join(self.build_dir, "CMakeCache.txt")
1848 dt_config_path = os.path.join(self.build_dir, "zephyr", _generated_dt_confg)
1849 defconfig_path = os.path.join(self.build_dir, "zephyr", ".config")
1850
1851 with open(defconfig_path, "r") as fp:
1852 defconfig = {}
1853 for line in fp.readlines():
1854 m = self.config_re.match(line)
1855 if not m:
1856 if line.strip() and not line.startswith("#"):
1857 sys.stderr.write("Unrecognized line %s\n" % line)
1858 continue
1859 defconfig[m.group(1)] = m.group(2).strip()
1860
1861 self.defconfig = defconfig
1862
1863 cmake_conf = {}
1864 try:
1865 cache = CMakeCache.from_file(cmake_cache_path)
1866 except FileNotFoundError:
1867 cache = {}
1868
1869 for k in iter(cache):
1870 cmake_conf[k.name] = k.value
1871
1872 self.cmake_cache = cmake_conf
1873
1874 dt_conf = {}
1875 if os.path.exists(dt_config_path):
1876 with open(dt_config_path, "r") as fp:
1877 for line in fp.readlines():
1878 m = self.dt_re.match(line)
1879 if not m:
1880 if line.strip() and not line.startswith("#"):
1881 sys.stderr.write("Unrecognized line %s\n" % line)
1882 continue
1883 dt_conf[m.group(1)] = m.group(2).strip()
1884 self.devicetree = dt_conf
1885
1886 filter_data = {
1887 "ARCH": self.platform.arch,
1888 "PLATFORM": self.platform.name
1889 }
1890 filter_data.update(os.environ)
1891 filter_data.update(self.defconfig)
1892 filter_data.update(self.cmake_cache)
1893 filter_data.update(self.devicetree)
1894
1895 if self.testcase and self.testcase.tc_filter:
1896 try:
1897 res = expr_parser.parse(self.testcase.tc_filter, filter_data)
1898 except (ValueError, SyntaxError) as se:
1899 sys.stderr.write(
1900 "Failed processing %s\n" % self.testcase.yamlfile)
1901 raise se
1902
1903 if not res:
1904 return {os.path.join(self.platform.name, self.testcase.name): True}
1905 else:
1906 return {os.path.join(self.platform.name, self.testcase.name): False}
1907 else:
1908 self.platform.filter_data = filter_data
1909 return filter_data
1910
1911
1912class ProjectBuilder(FilterBuilder):
1913
1914 def __init__(self, suite, instance):
1915 super().__init__(instance.testcase, instance.platform, instance.testcase.source_dir, instance.build_dir)
1916
1917 self.log = "build.log"
1918 self.instance = instance
1919 self.suite = suite
1920
1921 def setup_handler(self):
1922
1923 instance = self.instance
1924 args = []
1925
1926 # FIXME: Needs simplification
1927 if instance.platform.simulation == "qemu":
1928 instance.handler = QEMUHandler(instance, "qemu")
1929 args.append("QEMU_PIPE=%s" % instance.handler.get_fifo())
1930 instance.handler.call_make_run = True
1931 elif instance.testcase.type == "unit":
1932 instance.handler = BinaryHandler(instance, "unit")
1933 instance.handler.binary = os.path.join(instance.build_dir, "testbinary")
1934 elif instance.platform.type == "native":
1935 instance.handler = BinaryHandler(instance, "native")
1936 instance.handler.binary = os.path.join(instance.build_dir, "zephyr", "zephyr.exe")
1937 elif instance.platform.simulation == "nsim":
1938 if find_executable("nsimdrv"):
1939 instance.handler = BinaryHandler(instance, "nsim")
1940 instance.handler.call_make_run = True
1941 elif instance.platform.simulation == "renode":
1942 if find_executable("renode"):
1943 instance.handler = BinaryHandler(instance, "renode")
1944 instance.handler.pid_fn = os.path.join(instance.build_dir, "renode.pid")
1945 instance.handler.call_make_run = True
1946 elif options.device_testing:
1947 instance.handler = DeviceHandler(instance, "device")
1948
1949 if instance.handler:
1950 instance.handler.args = args
1951
1952 def process(self, message):
1953 op = message.get('op')
1954
1955 if not self.instance.handler:
1956 self.setup_handler()
1957
1958 # The build process, call cmake and build with configured generator
1959 if op == "cmake":
1960 results = self.cmake()
1961 if self.instance.status == "failed":
1962 pipeline.put({"op": "report", "test": self.instance})
1963 elif options.cmake_only:
1964 pipeline.put({"op": "report", "test": self.instance})
1965 else:
1966 if self.instance.name in results['filter'] and results['filter'][self.instance.name]:
1967 verbose("filtering %s" % self.instance.name)
1968 self.instance.status = "skipped"
1969 self.instance.reason = "filter"
1970 pipeline.put({"op": "report", "test": self.instance})
1971 else:
1972 pipeline.put({"op": "build", "test": self.instance})
1973
1974
1975 elif op == "build":
1976 verbose("build test: %s" %self.instance.name)
1977 results = self.build()
1978
1979 if results.get('returncode', 1) > 0:
1980 pipeline.put({"op": "report", "test": self.instance})
1981 else:
1982 if self.instance.run:
1983 pipeline.put({"op": "run", "test": self.instance})
1984 else:
1985 pipeline.put({"op": "report", "test": self.instance})
1986 # Run the generated binary using one of the supported handlers
1987 elif op == "run":
1988 verbose("run test: %s" %self.instance.name)
1989 self.run()
1990 self.instance.status, _ = self.instance.handler.get_state()
1991 self.instance.reason = ""
1992 pipeline.put({
1993 "op": "report",
1994 "test": self.instance,
1995 "state": "executed",
1996 "status": self.instance.status,
1997 "reason": self.instance.status}
1998 )
1999
2000 # Report results and output progress to screen
2001 elif op == "report":
2002 self.report_out()
2003
2004 def report_out(self):
2005 total_tests_width = len(str(self.suite.total_tests))
2006 self.suite.total_done += 1
2007 instance = self.instance
2008
2009 if instance.status in ["failed", "timeout"]:
2010 self.suite.total_failed += 1
2011 if VERBOSE or not TERMINAL:
2012 status = COLOR_RED + "FAILED " + COLOR_NORMAL + instance.reason
2013 else:
2014 info(
2015 "{:<25} {:<50} {}FAILED{}: {}".format(
2016 instance.platform.name,
2017 instance.testcase.name,
2018 COLOR_RED,
2019 COLOR_NORMAL,
2020 instance.reason), False)
2021
2022 # FIXME
2023 h_log = "{}/handler.log".format(instance.build_dir)
2024 b_log = "{}/build.log".format(instance.build_dir)
2025 if os.path.exists(h_log):
2026 log_info("{}".format(h_log))
2027 else:
2028 log_info("{}".format(b_log))
2029
2030 elif instance.status == "skipped":
2031 self.suite.total_skipped += 1
2032 status = COLOR_YELLOW + "SKIPPED" + COLOR_NORMAL
2033
2034 else:
2035 status = COLOR_GREEN + "PASSED" + COLOR_NORMAL
2036
2037 if VERBOSE or not TERMINAL:
2038 if options.cmake_only:
2039 more_info = "cmake"
2040 elif instance.status == "skipped":
2041 more_info = instance.reason
2042 else:
2043 if instance.handler and instance.run:
2044 more_info = instance.handler.type_str
2045 htime = instance.handler.duration
2046 if htime:
2047 more_info += " {:.3f}s".format(htime)
2048 else:
2049 more_info = "build"
2050
2051 info("{:>{}}/{} {:<25} {:<50} {} ({})".format(
2052 self.suite.total_done, total_tests_width, self.suite.total_tests, instance.platform.name,
2053 instance.testcase.name, status, more_info))
2054
2055 if instance.status in ["failed", "timeout"]:
2056 h_log = "{}/handler.log".format(instance.build_dir)
2057 b_log = "{}/build.log".format(instance.build_dir)
2058 if os.path.exists(h_log):
2059 log_info("{}".format(h_log))
2060 else:
2061 log_info("{}".format(b_log))
2062
2063 else:
2064 sys.stdout.write("\rtotal complete: %s%4d/%4d%s %2d%% skipped: %s%4d%s, failed: %s%4d%s" % (
2065 COLOR_GREEN,
2066 self.suite.total_done,
2067 self.suite.total_tests,
2068 COLOR_NORMAL,
2069 int((float(self.suite.total_done) / self.suite.total_tests) * 100),
2070 COLOR_YELLOW if self.suite.total_skipped > 0 else COLOR_NORMAL,
2071 self.suite.total_skipped,
2072 COLOR_NORMAL,
2073 COLOR_RED if self.suite.total_failed > 0 else COLOR_NORMAL,
2074 self.suite.total_failed,
2075 COLOR_NORMAL
2076 )
2077 )
2078 sys.stdout.flush()
2079
2080 def cmake(self):
2081
2082 instance = self.instance
2083 args = self.testcase.extra_args[:]
2084
2085 if options.extra_args:
2086 args += options.extra_args
2087
2088 if instance.handler:
2089 args += instance.handler.args
2090
2091 # merge overlay files into one variable
2092 overlays = ""
2093 idx = 0
2094 for arg in args:
2095 match = re.search('OVERLAY_CONFIG="(.*)"', arg)
2096 if match:
2097 overlays += match.group(1)
2098 del args[idx]
2099 idx += 1
2100
2101 if self.testcase.extra_configs or options.coverage:
2102 args.append("OVERLAY_CONFIG=\"%s %s\"" %(overlays,
2103 os.path.join(instance.build_dir,
2104 "sanitycheck", "testcase_extra.conf")))
2105
2106 results = self.run_cmake(args)
2107 return results
2108
2109 def build(self):
2110 results = self.run_build(['--build', self.build_dir])
2111 return results
2112
2113 def run(self):
2114
2115 instance = self.instance
2116
2117 if instance.handler.type_str == "device":
2118 instance.handler.suite = self.suite
2119
2120 instance.handler.handle()
2121
2122 if instance.handler.type_str == "qemu":
2123 verbose("Running %s (%s)" %(instance.name, instance.handler.type_str))
2124 command = [get_generator()[0]]
2125 command += ["-C", self.build_dir, "run"]
2126
2127 with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.build_dir) as proc:
2128 verbose("Spawning QEMUHandler Thread for %s" % instance.name)
2129 proc.wait()
2130 self.returncode = proc.returncode
2131
2132 sys.stdout.flush()
2133
2134
2135pipeline = queue.LifoQueue()
2136
2137class BoundedExecutor(concurrent.futures.ThreadPoolExecutor):
2138 """BoundedExecutor behaves as a ThreadPoolExecutor which will block on
2139 calls to submit() once the limit given as "bound" work items are queued for
2140 execution.
2141 :param bound: Integer - the maximum number of items in the work queue
2142 :param max_workers: Integer - the size of the thread pool
2143 """
2144 def __init__(self, bound, max_workers, **kwargs):
2145 super().__init__(max_workers)
2146 #self.executor = ThreadPoolExecutor(max_workers=max_workers)
2147 self.semaphore = BoundedSemaphore(bound + max_workers)
2148
2149 def submit(self, fn, *args, **kwargs):
2150 self.semaphore.acquire()
2151 try:
2152 future = super().submit(fn, *args, **kwargs)
2153 except:
2154 self.semaphore.release()
2155 raise
2156 else:
2157 future.add_done_callback(lambda x: self.semaphore.release())
2158 return future
2159
2160 def shutdown(self, wait=True):
2161 super().shutdown(wait)
Andrew Boie4ef16c52015-08-28 12:36:03 -07002162
Andrew Boie6acbe632015-07-17 12:03:52 -07002163
2164class TestSuite:
Andrew Boie60823c22017-02-10 09:43:07 -08002165 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Anas Nashif1c65b6b2018-12-02 19:12:21 -05002166 dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Andrew Boie6acbe632015-07-17 12:03:52 -07002167
Anas Nashif83fc06a2019-06-22 11:04:10 -04002168 tc_schema = scl.yaml_load(
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +03002169 os.path.join(ZEPHYR_BASE,
Anas Nashif83fc06a2019-06-22 11:04:10 -04002170 "scripts", "sanity_chk", "testcase-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07002171
Anas Nashif37f9dc52018-02-23 08:53:46 -06002172 def __init__(self, board_root_list, testcase_roots, outdir):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002173
2174 self.roots = testcase_roots
2175 if not isinstance(board_root_list, list):
2176 self.board_roots= [board_root_list]
2177 else:
2178 self.board_roots = board_root_list
2179
Andrew Boie6acbe632015-07-17 12:03:52 -07002180 # Keep track of which test cases we've filtered out and why
Andrew Boie6acbe632015-07-17 12:03:52 -07002181 self.testcases = {}
2182 self.platforms = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04002183 self.default_platforms = []
Andrew Boieb391e662015-08-31 15:25:45 -07002184 self.outdir = os.path.abspath(outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07002185 self.discards = None
Kumar Galac84235e2018-04-10 13:32:51 -05002186 self.load_errors = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04002187 self.instances = dict()
Andrew Boie6acbe632015-07-17 12:03:52 -07002188
Anas Nashif83fc06a2019-06-22 11:04:10 -04002189 self.total_tests = 0 # number of test instances
2190 self.total_cases = 0 # number of test cases
2191 self.total_done = 0 # tests completed
2192 self.total_failed = 0
2193 self.total_skipped = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07002194
Anas Nashif83fc06a2019-06-22 11:04:10 -04002195 self.total_platforms = 0
2196 self.start_time = 0
2197 self.duration = 0
2198 self.warnings = 0
2199 self.cv = threading.Condition()
Anas Nashif61e21632018-04-08 13:30:16 -05002200
Anas Nashif83fc06a2019-06-22 11:04:10 -04002201 # hardcoded for now
2202 self.connected_hardware = []
Andrew Boie3d348712016-04-08 11:52:13 -07002203
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07002204
Anas Nashif83fc06a2019-06-22 11:04:10 -04002205 if options.jobs:
2206 self.jobs = options.jobs
2207 elif options.build_only:
2208 self.jobs = multiprocessing.cpu_count() * 2
Andy Ross9c9162d2019-01-03 10:50:53 -08002209 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002210 self.jobs = multiprocessing.cpu_count()
Daniel Leung6b170072016-04-07 12:10:25 -07002211
Anas Nashif83fc06a2019-06-22 11:04:10 -04002212 info("JOBS: %d" % self.jobs)
Andrew Boie6acbe632015-07-17 12:03:52 -07002213
Anas Nashif83fc06a2019-06-22 11:04:10 -04002214 def update(self):
2215 self.total_tests = len(self.instances)
2216 self.total_cases = len(self.testcases)
Anas Nashifbd166f42017-09-02 12:32:08 -04002217
Andrew Boie6acbe632015-07-17 12:03:52 -07002218
2219 def compare_metrics(self, filename):
2220 # name, datatype, lower results better
2221 interesting_metrics = [("ram_size", int, True),
2222 ("rom_size", int, True)]
2223
Andrew Boie6acbe632015-07-17 12:03:52 -07002224
2225 if not os.path.exists(filename):
2226 info("Cannot compare metrics, %s not found" % filename)
2227 return []
2228
2229 results = []
2230 saved_metrics = {}
2231 with open(filename) as fp:
2232 cr = csv.DictReader(fp)
2233 for row in cr:
2234 d = {}
2235 for m, _, _ in interesting_metrics:
2236 d[m] = row[m]
2237 saved_metrics[(row["test"], row["platform"])] = d
2238
Anas Nashif83fc06a2019-06-22 11:04:10 -04002239 for instance in self.instances.values():
2240 mkey = (instance.testcase.name, instance.platform.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07002241 if mkey not in saved_metrics:
2242 continue
2243 sm = saved_metrics[mkey]
2244 for metric, mtype, lower_better in interesting_metrics:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002245 if metric not in instance.metrics:
Andrew Boie6acbe632015-07-17 12:03:52 -07002246 continue
2247 if sm[metric] == "":
2248 continue
Anas Nashif83fc06a2019-06-22 11:04:10 -04002249 delta = instance.metrics.get(metric, 0) - mtype(sm[metric])
Andrew Boieea7928f2015-08-14 14:27:38 -07002250 if delta == 0:
2251 continue
Anas Nashif83fc06a2019-06-22 11:04:10 -04002252 results.append((instance, metric, instance.metrics.get(metric, 0 ), delta,
Andrew Boieea7928f2015-08-14 14:27:38 -07002253 lower_better))
Andrew Boie6acbe632015-07-17 12:03:52 -07002254 return results
2255
Anas Nashif83fc06a2019-06-22 11:04:10 -04002256 def misc_reports(self, report, show_footprint, all_deltas,
2257 footprint_threshold, last_metrics):
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002258
Anas Nashif83fc06a2019-06-22 11:04:10 -04002259 if not report:
2260 return
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002261
Anas Nashif83fc06a2019-06-22 11:04:10 -04002262 deltas = self.compare_metrics(report)
2263 warnings = 0
2264 if deltas and show_footprint:
2265 for i, metric, value, delta, lower_better in deltas:
2266 if not all_deltas and ((delta < 0 and lower_better) or
2267 (delta > 0 and not lower_better)):
2268 continue
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002269
Anas Nashif83fc06a2019-06-22 11:04:10 -04002270 percentage = (float(delta) / float(value - delta))
2271 if not all_deltas and (percentage <
2272 (footprint_threshold / 100.0)):
2273 continue
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002274
Anas Nashif83fc06a2019-06-22 11:04:10 -04002275 info("{:<25} {:<60} {}{}{}: {} {:<+4}, is now {:6} {:+.2%}".format(
2276 i.platform.name, i.testcase.name, COLOR_YELLOW,
2277 "INFO" if all_deltas else "WARNING", COLOR_NORMAL,
2278 metric, delta, value, percentage))
2279 warnings += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002280
Anas Nashif83fc06a2019-06-22 11:04:10 -04002281 if warnings:
2282 info("Deltas based on metrics from last %s" %
2283 ("release" if not last_metrics else "run"))
Anas Nashif61e21632018-04-08 13:30:16 -05002284
Anas Nashif83fc06a2019-06-22 11:04:10 -04002285 def summary(self, unrecognized_sections):
2286 failed = 0
2287 for instance in self.instances.values():
2288 if instance.status == "failed":
2289 failed += 1
2290 elif instance.metrics.get("unrecognized") and not unrecognized_sections:
2291 info("%sFAILED%s: %s has unrecognized binary sections: %s" %
2292 (COLOR_RED, COLOR_NORMAL, instance.name,
2293 str(instance.metrics.get("unrecognized", []))))
2294 failed += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002295
Anas Nashif83fc06a2019-06-22 11:04:10 -04002296 if self.total_tests and self.total_tests != self.total_skipped:
2297 pass_rate = (float(self.total_tests - self.total_failed - self.total_skipped)/ float(self.total_tests - self.total_skipped))
2298 else:
2299 pass_rate = 0
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002300
Anas Nashif83fc06a2019-06-22 11:04:10 -04002301 info("{}{} of {}{} tests passed ({:.2%}), {}{}{} failed, {} skipped with {}{}{} warnings in {:.2f} seconds".format(
2302 COLOR_RED if failed else COLOR_GREEN,
2303 self.total_tests - self.total_failed - self.total_skipped,
2304 self.total_tests,
2305 COLOR_NORMAL,
2306 pass_rate,
2307 COLOR_RED if self.total_failed else COLOR_NORMAL,
2308 self.total_failed,
2309 COLOR_NORMAL,
2310 self.total_skipped,
2311 COLOR_YELLOW if self.warnings else COLOR_NORMAL,
2312 self.warnings,
2313 COLOR_NORMAL,
2314 self.duration))
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002315
Anas Nashif83fc06a2019-06-22 11:04:10 -04002316 platforms = set(p.platform for p in self.instances.values())
2317 self.total_platforms = len(self.platforms)
2318 if self.platforms:
2319 info("In total {} test cases were executed on {} out of total {} platforms ({:02.2f}%)".format(
2320 self.total_cases,
2321 len(platforms),
2322 self.total_platforms,
2323 (100 * len(platforms) / len(self.platforms))
2324 ))
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002325
Anas Nashif83fc06a2019-06-22 11:04:10 -04002326 def save_reports(self):
2327 if not self.instances:
2328 return
Anas Nashif61e21632018-04-08 13:30:16 -05002329
Anas Nashif83fc06a2019-06-22 11:04:10 -04002330 report_name = "sanitycheck"
2331 if options.report_name:
2332 report_name = options.report_name
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002333
Anas Nashif83fc06a2019-06-22 11:04:10 -04002334 if options.report_dir:
Paul Sokolovsky3ea18692019-10-15 17:18:39 +03002335 os.makedirs(options.report_dir, exist_ok=True)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002336 filename = os.path.join(options.report_dir, report_name)
2337 outdir = options.report_dir
2338 else:
2339 filename = os.path.join(options.outdir, report_name)
2340 outdir = options.outdir
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002341
Anas Nashif83fc06a2019-06-22 11:04:10 -04002342 if not options.no_update:
2343 self.xunit_report(filename + ".xml")
2344 self.csv_report(filename + ".csv")
2345 self.target_report(outdir)
2346 if self.discards:
2347 self.discard_report(filename + "_discard.csv")
2348
2349 if options.release:
2350 self.csv_report(RELEASE_DATA)
2351
2352 if log_file:
2353 log_file.close()
2354
2355 def load_hardware_map_from_cmdline(self, serial, platform):
2356 device = {
2357 "serial": serial,
2358 "platform": platform,
2359 "counter": 0,
2360 "available": True,
2361 "connected": True
2362 }
2363 self.connected_hardware = [device]
2364
2365 def load_hardware_map(self, map_file):
2366 with open(map_file, 'r') as stream:
2367 try:
2368 self.connected_hardware = yaml.safe_load(stream)
2369 except yaml.YAMLError as exc:
2370 print(exc)
2371 for i in self.connected_hardware:
2372 i['counter'] = 0
2373
2374 def add_configurations(self):
2375
2376 for board_root in self.board_roots:
2377 board_root = os.path.abspath(board_root)
2378
2379 debug("Reading platform configuration files under %s..." %
2380 board_root)
2381
2382 for file in glob.glob(os.path.join(board_root, "*", "*", "*.yaml")):
2383 verbose("Found plaform configuration " + file)
2384 try:
2385 platform = Platform()
2386 platform.load(file)
2387 if platform.sanitycheck:
2388 self.platforms.append(platform)
2389 if platform.default:
2390 self.default_platforms.append(platform.name)
2391
2392 except RuntimeError as e:
2393 error("E: %s: can't load: %s" % (file, e))
2394 self.load_errors += 1
2395
2396 @staticmethod
2397 def get_toolchain():
2398 toolchain = os.environ.get("ZEPHYR_TOOLCHAIN_VARIANT", None) or \
2399 os.environ.get("ZEPHYR_GCC_VARIANT", None)
2400
2401 if toolchain == "gccarmemb":
2402 # Remove this translation when gccarmemb is no longer supported.
2403 toolchain = "gnuarmemb"
2404
Anas Nashifb4bdd662018-08-15 17:12:28 -05002405 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002406 if not toolchain:
2407 raise SanityRuntimeError("E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined")
Anas Nashifb4bdd662018-08-15 17:12:28 -05002408 except Exception as e:
2409 print(str(e))
2410 sys.exit(2)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002411
Anas Nashif83fc06a2019-06-22 11:04:10 -04002412 return toolchain
2413
2414
2415 def add_testcases(self):
2416 for root in self.roots:
2417 root = os.path.abspath(root)
2418
2419 debug("Reading test case configuration files under %s..." %root)
2420
2421 for dirpath, dirnames, filenames in os.walk(root, topdown=True):
2422 verbose("scanning %s" % dirpath)
2423 if 'sample.yaml' in filenames:
2424 filename = 'sample.yaml'
2425 elif 'testcase.yaml' in filenames:
2426 filename = 'testcase.yaml'
2427 else:
2428 continue
2429
2430 verbose("Found possible test case in " + dirpath)
2431
2432 dirnames[:] = []
2433 tc_path = os.path.join(dirpath, filename)
2434 self.add_testcase(tc_path, root)
2435
2436 def add_testcase(self, tc_data_file, root):
2437 try:
2438 parsed_data = SanityConfigParser(tc_data_file, self.tc_schema)
2439 parsed_data.load()
2440
2441 tc_path = os.path.dirname(tc_data_file)
2442 workdir = os.path.relpath(tc_path, root)
2443
2444 for name in parsed_data.tests.keys():
2445 tc = TestCase()
2446 tc.name = tc.get_unique(root, workdir, name)
2447
2448 tc_dict = parsed_data.get_test(name, testcase_valid_keys)
2449
2450 tc.source_dir = tc_path
2451 tc.yamlfile = tc_data_file
2452
2453 tc.id = name
2454 tc.type = tc_dict["type"]
2455 tc.tags = tc_dict["tags"]
2456 tc.extra_args = tc_dict["extra_args"]
2457 tc.extra_configs = tc_dict["extra_configs"]
2458 tc.arch_whitelist = tc_dict["arch_whitelist"]
2459 tc.arch_exclude = tc_dict["arch_exclude"]
2460 tc.skip = tc_dict["skip"]
2461 tc.platform_exclude = tc_dict["platform_exclude"]
2462 tc.platform_whitelist = tc_dict["platform_whitelist"]
2463 tc.toolchain_exclude = tc_dict["toolchain_exclude"]
2464 tc.toolchain_whitelist = tc_dict["toolchain_whitelist"]
2465 tc.tc_filter = tc_dict["filter"]
2466 tc.timeout = tc_dict["timeout"]
2467 tc.harness = tc_dict["harness"]
2468 tc.harness_config = tc_dict["harness_config"]
2469 tc.build_only = tc_dict["build_only"]
2470 tc.build_on_all = tc_dict["build_on_all"]
2471 tc.slow = tc_dict["slow"]
2472 tc.min_ram = tc_dict["min_ram"]
2473 tc.depends_on = tc_dict["depends_on"]
2474 tc.min_flash = tc_dict["min_flash"]
2475 tc.extra_sections = tc_dict["extra_sections"]
2476
2477 tc.parse_subcases(tc_path)
2478
2479 if tc.name:
2480 self.testcases[tc.name] = tc
2481
2482 except Exception as e:
2483 error("E: %s: can't load (skipping): %s" % (tc_data_file, e))
2484 self.load_errors += 1
2485 return False
2486
2487 return True
2488
2489
2490 def get_platform(self, name):
2491 selected_platform = None
2492 for platform in self.platforms:
2493 if platform.name == name:
2494 selected_platform = platform
2495 break
2496 return selected_platform
2497
2498 def get_last_failed(self):
2499 last_run = os.path.join(options.outdir, "sanitycheck.csv")
2500 try:
2501 if not os.path.exists(last_run):
2502 raise SanityRuntimeError("Couldn't find last sanitycheck run.: %s" %last_run)
2503 except Exception as e:
2504 print(str(e))
2505 sys.exit(2)
2506
2507 total_tests = 0
2508 with open(last_run, "r") as fp:
2509 cr = csv.DictReader(fp)
2510 instance_list = []
2511 for row in cr:
2512 total_tests += 1
2513 if row["passed"] == "True":
2514 continue
2515 test = row["test"]
2516 platform = self.get_platform(row["platform"])
2517 instance = TestInstance(self.testcases[test], platform, self.outdir)
2518 instance.create_overlay(platform.name)
2519 instance_list.append(instance)
2520 self.add_instances(instance_list)
2521
2522 tests_to_run = len(self.instances)
2523 info("%d tests passed already, retyring %d tests" %(total_tests - tests_to_run, tests_to_run))
2524
2525 def load_from_file(self, file):
2526 try:
2527 if not os.path.exists(file):
2528 raise SanityRuntimeError(
2529 "Couldn't find input file with list of tests.")
2530 except Exception as e:
2531 print(str(e))
2532 sys.exit(2)
2533
2534 with open(file, "r") as fp:
2535 cr = csv.DictReader(fp)
2536 instance_list = []
2537 for row in cr:
2538 if row["arch"] == "arch":
2539 continue
2540 test = row["test"]
2541 platform = self.get_platform(row["platform"])
2542 instance = TestInstance(self.testcases[test], platform, self.outdir)
2543 instance.create_overlay(platform.name)
2544 instance_list.append(instance)
2545 self.add_instances(instance_list)
2546
2547
2548 def apply_filters(self):
2549
2550 toolchain = self.get_toolchain()
2551
2552 discards = {}
2553 platform_filter = options.platform
2554 testcase_filter = run_individual_tests
2555 arch_filter = options.arch
2556 tag_filter = options.tag
2557 exclude_tag = options.exclude_tag
2558
2559 verbose("platform filter: " + str(platform_filter))
2560 verbose(" arch_filter: " + str(arch_filter))
2561 verbose(" tag_filter: " + str(tag_filter))
2562 verbose(" exclude_tag: " + str(exclude_tag))
2563
2564 default_platforms = False
2565
2566 if platform_filter:
2567 platforms = list(filter(lambda p: p.name in platform_filter, self.platforms))
2568 else:
2569 platforms = self.platforms
2570
2571 if options.all:
2572 info("Selecting all possible platforms per test case")
2573 # When --all used, any --platform arguments ignored
2574 platform_filter = []
2575 elif not platform_filter:
2576 info("Selecting default platforms per test case")
2577 default_platforms = True
2578
2579 info("Building initial testcase list...")
2580
2581 for tc_name, tc in self.testcases.items():
2582 # list of instances per testcase, aka configurations.
2583 instance_list = []
2584 for plat in platforms:
2585 instance = TestInstance(tc, plat, self.outdir)
2586
2587 if (plat.arch == "unit") != (tc.type == "unit"):
2588 # Discard silently
2589 continue
2590
2591 if options.device_testing and instance.build_only:
2592 discards[instance] = "Not runnable on device"
2593 continue
2594
2595 if tc.skip:
2596 discards[instance] = "Skip filter"
2597 continue
2598
2599 if tc.build_on_all and not platform_filter:
2600 platform_filter = []
2601
2602 if tag_filter and not tc.tags.intersection(tag_filter):
2603 discards[instance] = "Command line testcase tag filter"
2604 continue
2605
2606 if exclude_tag and tc.tags.intersection(exclude_tag):
2607 discards[instance] = "Command line testcase exclude filter"
2608 continue
2609
2610 if testcase_filter and tc_name not in testcase_filter:
2611 discards[instance] = "Testcase name filter"
2612 continue
2613
2614 if arch_filter and plat.arch not in arch_filter:
2615 discards[instance] = "Command line testcase arch filter"
2616 continue
2617
2618 if tc.arch_whitelist and plat.arch not in tc.arch_whitelist:
2619 discards[instance] = "Not in test case arch whitelist"
2620 continue
2621
2622 if tc.arch_exclude and plat.arch in tc.arch_exclude:
2623 discards[instance] = "In test case arch exclude"
2624 continue
2625
2626 if tc.platform_exclude and plat.name in tc.platform_exclude:
2627 discards[instance] = "In test case platform exclude"
2628 continue
2629
2630 if tc.toolchain_exclude and toolchain in tc.toolchain_exclude:
2631 discards[instance] = "In test case toolchain exclude"
2632 continue
2633
2634 if platform_filter and plat.name not in platform_filter:
2635 discards[instance] = "Command line platform filter"
2636 continue
2637
2638 if tc.platform_whitelist and plat.name not in tc.platform_whitelist:
2639 discards[instance] = "Not in testcase platform whitelist"
2640 continue
2641
2642 if tc.toolchain_whitelist and toolchain not in tc.toolchain_whitelist:
2643 discards[instance] = "Not in testcase toolchain whitelist"
2644 continue
2645
2646 if not plat.env_satisfied:
2647 discards[instance] = "Environment ({}) not satisfied".format(", ".join(plat.env))
2648 continue
2649
2650 if not options.force_toolchain \
2651 and toolchain and (toolchain not in plat.supported_toolchains) \
2652 and tc.type != 'unit':
2653 discards[instance] = "Not supported by the toolchain"
2654 continue
2655
2656 if plat.ram < tc.min_ram:
2657 discards[instance] = "Not enough RAM"
2658 continue
2659
2660 if tc.depends_on:
2661 dep_intersection = tc.depends_on.intersection(set(plat.supported))
2662 if dep_intersection != set(tc.depends_on):
2663 discards[instance] = "No hardware support"
2664 continue
2665
2666 if plat.flash < tc.min_flash:
2667 discards[instance] = "Not enough FLASH"
2668 continue
2669
2670 if set(plat.ignore_tags) & tc.tags:
2671 discards[instance] = "Excluded tags per platform"
2672 continue
2673
2674 # if nothing stopped us until now, it means this configuration
2675 # needs to be added.
2676 instance_list.append(instance)
2677
2678 # no configurations, so jump to next testcase
2679 if not instance_list:
2680 continue
2681
2682 # if sanitycheck was launched with no platform options at all, we
2683 # take all default platforms
2684 if default_platforms and not tc.build_on_all:
2685 if tc.platform_whitelist:
2686 a = set(self.default_platforms)
2687 b = set(tc.platform_whitelist)
2688 c = a.intersection(b)
2689 if c:
2690 aa = list( filter( lambda tc: tc.platform.name in c, instance_list))
2691 self.add_instances(aa)
2692 else:
2693 self.add_instances(instance_list[:1])
2694 else:
2695 instances = list( filter( lambda tc: tc.platform.default, instance_list))
2696 self.add_instances(instances)
2697
2698 for instance in list(filter(lambda tc: not tc.platform.default, instance_list)):
2699 discards[instance] = "Not a default test platform"
2700
2701 else:
2702 self.add_instances(instance_list)
2703
2704 for _, case in self.instances.items():
2705 case.create_overlay(case.platform.name)
2706
2707 self.discards = discards
2708
2709 return discards
2710
2711 def add_instances(self, instance_list):
2712 for instance in instance_list:
2713 self.instances[instance.name] = instance
2714
2715 def add_tasks_to_queue(self):
2716 for instance in self.instances.values():
2717 if options.test_only:
2718 if instance.run:
2719 pipeline.put({"op": "run", "test": instance, "status": "built"})
2720 else:
2721 if instance.status not in ['passed', 'skipped']:
2722 instance.status = None
2723 pipeline.put({"op": "cmake", "test": instance})
2724
2725 return "DONE FEEDING"
2726
2727 def execute(self):
2728 def calc_one_elf_size(instance):
2729 if instance.status not in ["failed", "skipped"]:
2730 if instance.platform.type != "native":
2731 size_calc = instance.calculate_sizes()
2732 instance.metrics["ram_size"] = size_calc.get_ram_size()
2733 instance.metrics["rom_size"] = size_calc.get_rom_size()
2734 instance.metrics["unrecognized"] = size_calc.unrecognized_sections()
2735 else:
2736 instance.metrics["ram_size"] = 0
2737 instance.metrics["rom_size"] = 0
2738 instance.metrics["unrecognized"] = []
2739
2740 instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0
2741
2742 info("Adding tasks to the queue...")
2743 # We can use a with statement to ensure threads are cleaned up promptly
2744 with BoundedExecutor(bound=self.jobs, max_workers=self.jobs) as executor:
2745
2746 # start a future for a thread which sends work in through the queue
2747 future_to_test = {
2748 executor.submit(self.add_tasks_to_queue): 'FEEDER DONE'}
2749
2750 while future_to_test:
2751 # check for status of the futures which are currently working
2752 done, _ = concurrent.futures.wait(
2753 future_to_test, timeout=0.25,
2754 return_when=concurrent.futures.FIRST_COMPLETED)
2755
2756 # if there is incoming work, start a new future
2757 while not pipeline.empty():
2758 # fetch a url from the queue
2759 message = pipeline.get()
2760 test = message['test']
2761
2762 # Start the load operation and mark the future with its URL
2763 pb = ProjectBuilder(self, test)
2764 future_to_test[executor.submit(pb.process, message)] = test.name
2765
2766 # process any completed futures
2767 for future in done:
2768 test = future_to_test[future]
2769 try:
2770 data = future.result()
2771 except Exception as exc:
2772 print('%r generated an exception: %s' % (test, exc))
2773 else:
2774 if data:
2775 verbose(data)
2776
2777 # remove the now completed future
2778 del future_to_test[future]
2779
2780 if options.enable_size_report and not options.cmake_only:
2781 # Parallelize size calculation
2782 executor = concurrent.futures.ThreadPoolExecutor(self.jobs)
2783 futures = [executor.submit(calc_one_elf_size, instance)
2784 for instance in self.instances.values()]
2785 concurrent.futures.wait(futures)
2786 else:
2787 for instance in self.instances.values():
2788 instance.metrics["ram_size"] = 0
2789 instance.metrics["rom_size"] = 0
2790 instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0
2791 instance.metrics["unrecognized"] = []
2792
2793
2794 def discard_report(self, filename):
2795
2796 try:
2797 if self.discards is None:
2798 raise SanityRuntimeError("apply_filters() hasn't been run!")
2799 except Exception as e:
2800 error(str(e))
2801 sys.exit(2)
2802
2803 with open(filename, "wt") as csvfile:
2804 fieldnames = ["test", "arch", "platform", "reason"]
2805 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2806 cw.writeheader()
2807 for instance, reason in sorted(self.discards.items()):
2808 rowdict = {"test": instance.testcase.name,
2809 "arch": instance.platform.arch,
2810 "platform": instance.platform.name,
2811 "reason": reason}
2812 cw.writerow(rowdict)
2813
2814
2815 def target_report(self, outdir):
2816 run = "Sanitycheck"
2817 eleTestsuite = None
2818
2819 platforms = {inst.platform.name for _,inst in self.instances.items()}
2820 for platform in platforms:
2821 errors = 0
2822 passes = 0
2823 fails = 0
2824 duration = 0
2825 skips = 0
2826 for _, instance in self.instances.items():
2827 if instance.platform.name != platform:
2828 continue
2829
2830 handler_time = instance.metrics.get('handler_time', 0)
2831 duration += handler_time
2832 for k in instance.results.keys():
2833 if instance.results[k] == 'PASS':
2834 passes += 1
2835 elif instance.results[k] == 'BLOCK':
2836 errors += 1
2837 elif instance.results[k] == 'SKIP':
2838 skips += 1
2839 else:
2840 fails += 1
2841
2842 eleTestsuites = ET.Element('testsuites')
2843 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
2844 name=run, time="%f" % duration,
2845 tests="%d" % (errors + passes + fails),
2846 failures="%d" % fails,
2847 errors="%d" % errors, skipped="%d" %skips)
2848
2849 handler_time = 0
2850
2851 # print out test results
2852 for _, instance in self.instances.items():
2853 if instance.platform.name != platform:
2854 continue
2855 handler_time = instance.metrics.get('handler_time', 0)
2856 for k in instance.results.keys():
2857 eleTestcase = ET.SubElement(
2858 eleTestsuite, 'testcase', classname="%s:%s" %(instance.platform.name, os.path.basename(instance.testcase.name)),
2859 name="%s" % (k), time="%f" %handler_time)
2860 if instance.results[k] in ['FAIL', 'BLOCK']:
2861 el = None
2862
2863 if instance.results[k] == 'FAIL':
2864 el = ET.SubElement(
2865 eleTestcase,
2866 'failure',
2867 type="failure",
2868 message="failed")
2869 elif instance.results[k] == 'BLOCK':
2870 el = ET.SubElement(
2871 eleTestcase,
2872 'error',
2873 type="failure",
2874 message="failed")
2875 p = os.path.join(options.outdir, instance.platform.name, instance.testcase.name)
2876 log_file = os.path.join(p, "handler.log")
2877
2878 if os.path.exists(log_file):
2879 with open(log_file, "rb") as f:
2880 log = f.read().decode("utf-8")
2881 filtered_string = ''.join(filter(lambda x: x in string.printable, log))
2882 el.text = filtered_string
2883
2884 elif instance.results[k] == 'SKIP':
2885 el = ET.SubElement(
2886 eleTestcase,
2887 'skipped',
2888 type="skipped",
2889 message="Skipped")
2890
2891
2892 result = ET.tostring(eleTestsuites)
2893 with open(os.path.join(outdir, platform + ".xml"), 'wb') as f:
2894 f.write(result)
2895
2896
2897 def xunit_report(self, filename):
Anas Nashifb3311ed2017-04-13 14:44:48 -04002898 fails = 0
2899 passes = 0
2900 errors = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04002901 skips = 0
2902 duration = 0
Anas Nashifb3311ed2017-04-13 14:44:48 -04002903
Anas Nashif83fc06a2019-06-22 11:04:10 -04002904 for instance in self.instances.values():
2905 handler_time = instance.metrics.get('handler_time', 0)
2906 duration += handler_time
2907 if instance.status == "failed":
2908 if instance.reason in ['build_error', 'handler_crash']:
Anas Nashifb3311ed2017-04-13 14:44:48 -04002909 errors += 1
2910 else:
2911 fails += 1
Anas Nashif83fc06a2019-06-22 11:04:10 -04002912 elif instance.status == 'skipped':
2913 skips += 1
Anas Nashifb3311ed2017-04-13 14:44:48 -04002914 else:
2915 passes += 1
2916
2917 run = "Sanitycheck"
2918 eleTestsuite = None
Anas Nashif4f028882017-12-30 11:48:43 -05002919 append = options.only_failed
Anas Nashifb3311ed2017-04-13 14:44:48 -04002920
Anas Nashif83fc06a2019-06-22 11:04:10 -04002921 # When we re-run the tests, we re-use the results and update only with
2922 # the newly run tests.
Anas Nashif0605fa32017-05-07 08:51:02 -04002923 if os.path.exists(filename) and append:
Anas Nashifb3311ed2017-04-13 14:44:48 -04002924 tree = ET.parse(filename)
2925 eleTestsuites = tree.getroot()
Anas Nashif3ba1d432017-12-05 15:28:44 -05002926 eleTestsuite = tree.findall('testsuite')[0]
Anas Nashifb3311ed2017-04-13 14:44:48 -04002927 else:
2928 eleTestsuites = ET.Element('testsuites')
Anas Nashif3ba1d432017-12-05 15:28:44 -05002929 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
Anas Nashif83fc06a2019-06-22 11:04:10 -04002930 name=run, time="%f" % duration,
2931 tests="%d" % (errors + passes + fails + skips),
Anas Nashif3ba1d432017-12-05 15:28:44 -05002932 failures="%d" % fails,
Anas Nashif83fc06a2019-06-22 11:04:10 -04002933 errors="%d" %(errors), skip="%s" %(skips))
Anas Nashifb3311ed2017-04-13 14:44:48 -04002934
Anas Nashif83fc06a2019-06-22 11:04:10 -04002935 for instance in self.instances.values():
Anas Nashifb3311ed2017-04-13 14:44:48 -04002936
Anas Nashif83fc06a2019-06-22 11:04:10 -04002937 # remove testcases that are a re-run
Anas Nashifb3311ed2017-04-13 14:44:48 -04002938 if append:
2939 for tc in eleTestsuite.findall('testcase'):
Anas Nashif3ba1d432017-12-05 15:28:44 -05002940 if tc.get('classname') == "%s:%s" % (
Anas Nashif83fc06a2019-06-22 11:04:10 -04002941 instance.platform.name, instance.testcase.name):
Anas Nashifb3311ed2017-04-13 14:44:48 -04002942 eleTestsuite.remove(tc)
2943
Anas Nashif83fc06a2019-06-22 11:04:10 -04002944 handler_time = 0
2945 if instance.status != "failed" and instance.handler:
2946 handler_time = instance.metrics.get("handler_time", 0)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002947
Anas Nashif3ba1d432017-12-05 15:28:44 -05002948 eleTestcase = ET.SubElement(
2949 eleTestsuite, 'testcase', classname="%s:%s" %
Anas Nashif83fc06a2019-06-22 11:04:10 -04002950 (instance.platform.name, instance.testcase.name), name="%s" %
2951 (instance.testcase.name), time="%f" %handler_time)
2952
2953 if instance.status == "failed":
Anas Nashif3ba1d432017-12-05 15:28:44 -05002954 failure = ET.SubElement(
2955 eleTestcase,
2956 'failure',
2957 type="failure",
Anas Nashif83fc06a2019-06-22 11:04:10 -04002958 message=instance.reason)
2959 p = ("%s/%s/%s" % (options.outdir, instance.platform.name, instance.testcase.name))
Anas Nashifb3311ed2017-04-13 14:44:48 -04002960 bl = os.path.join(p, "build.log")
Anas Nashifc96c90a2019-02-05 07:38:32 -05002961 hl = os.path.join(p, "handler.log")
2962 log_file = bl
Anas Nashif83fc06a2019-06-22 11:04:10 -04002963 if instance.reason != 'Build error':
Anas Nashifc96c90a2019-02-05 07:38:32 -05002964 if os.path.exists(hl):
2965 log_file = hl
2966 else:
2967 log_file = bl
Anas Nashifaccc8eb2017-05-01 16:33:43 -04002968
Anas Nashifc96c90a2019-02-05 07:38:32 -05002969 if os.path.exists(log_file):
2970 with open(log_file, "rb") as f:
Anas Nashif712d3452017-12-29 22:09:03 -05002971 log = f.read().decode("utf-8")
Anas Nashifa4c368e2018-10-15 09:45:59 -04002972 filtered_string = ''.join(filter(lambda x: x in string.printable, log))
2973 failure.text = filtered_string
Anas Nashifba4643b2018-09-23 09:41:59 -05002974 f.close()
Anas Nashif83fc06a2019-06-22 11:04:10 -04002975 elif instance.status == "skipped":
2976 ET.SubElement( eleTestcase, 'skipped', type="skipped", message="Skipped")
Anas Nashifb3311ed2017-04-13 14:44:48 -04002977
2978 result = ET.tostring(eleTestsuites)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002979 with open(filename, 'wb') as report:
2980 report.write(result)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002981
Andrew Boie6acbe632015-07-17 12:03:52 -07002982
Anas Nashif83fc06a2019-06-22 11:04:10 -04002983 def csv_report(self, filename):
Andrew Boie08ce5a52016-02-22 13:28:10 -08002984 with open(filename, "wt") as csvfile:
Andrew Boie6acbe632015-07-17 12:03:52 -07002985 fieldnames = ["test", "arch", "platform", "passed", "status",
Anas Nashif83fc06a2019-06-22 11:04:10 -04002986 "extra_args", "handler", "handler_time", "ram_size",
Andrew Boie6acbe632015-07-17 12:03:52 -07002987 "rom_size"]
2988 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2989 cw.writeheader()
Anas Nashif83fc06a2019-06-22 11:04:10 -04002990 for instance in sorted(self.instances.values()):
2991 rowdict = {"test": instance.testcase.name,
2992 "arch": instance.platform.arch,
2993 "platform": instance.platform.name,
2994 "extra_args": " ".join(instance.testcase.extra_args),
2995 "handler": instance.platform.simulation}
2996
2997 if instance.status in ["failed", "timeout"]:
Andrew Boie6acbe632015-07-17 12:03:52 -07002998 rowdict["passed"] = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04002999 rowdict["status"] = instance.reason
Andrew Boie6acbe632015-07-17 12:03:52 -07003000 else:
3001 rowdict["passed"] = True
Anas Nashif83fc06a2019-06-22 11:04:10 -04003002 if instance.handler:
3003 rowdict["handler_time"] = instance.metrics.get("handler_time", 0)
3004 ram_size = instance.metrics.get("ram_size", 0)
3005 rom_size = instance.metrics.get("rom_size", 0)
3006 rowdict["ram_size"] = ram_size
3007 rowdict["rom_size"] = rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -07003008 cw.writerow(rowdict)
3009
3010
3011def parse_arguments():
3012
Anas Nashif3ba1d432017-12-05 15:28:44 -05003013 parser = argparse.ArgumentParser(
3014 description=__doc__,
3015 formatter_class=argparse.RawDescriptionHelpFormatter)
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -05003016 parser.fromfile_prefix_chars = "+"
Andrew Boie6acbe632015-07-17 12:03:52 -07003017
Marc Herbert932a33a2019-03-12 11:37:53 -07003018 case_select = parser.add_argument_group("Test case selection",
3019 """
3020Artificially long but functional example:
3021 $ ./scripts/sanitycheck -v \\
Marc Herberte5cedca2019-04-08 14:02:34 -07003022 --testcase-root tests/ztest/base \\
3023 --testcase-root tests/kernel \\
Marc Herbert932a33a2019-03-12 11:37:53 -07003024 --test tests/ztest/base/testing.ztest.verbose_0 \\
3025 --test tests/kernel/fifo/fifo_api/kernel.fifo.poll
3026
3027 "kernel.fifo.poll" is one of the test section names in
3028 __/fifo_api/testcase.yaml
3029 """)
Marc Herbertedf17592019-03-08 12:39:11 -08003030
Anas Nashif07d54c02018-07-21 19:29:08 -05003031 parser.add_argument("--force-toolchain", action="store_true",
3032 help="Do not filter based on toolchain, use the set "
3033 " toolchain unconditionally")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003034 parser.add_argument(
3035 "-p", "--platform", action="append",
3036 help="Platform filter for testing. This option may be used multiple "
3037 "times. Testcases will only be built/run on the platforms "
3038 "specified. If this option is not used, then platforms marked "
3039 "as default in the platform metadata file will be chosen "
3040 "to build and test. ")
3041 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003042 "-a", "--arch", action="append",
3043 help="Arch filter for testing. Takes precedence over --platform. "
3044 "If unspecified, test all arches. Multiple invocations "
3045 "are treated as a logical 'or' relationship")
3046 parser.add_argument(
3047 "-t", "--tag", action="append",
3048 help="Specify tags to restrict which tests to run by tag value. "
3049 "Default is to not do any tag filtering. Multiple invocations "
3050 "are treated as a logical 'or' relationship")
Anas Nashifdfa86e22016-10-24 17:08:56 -04003051 parser.add_argument("-e", "--exclude-tag", action="append",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003052 help="Specify tags of tests that should not run. "
3053 "Default is to run all tests with all tags.")
Marc Herbertedf17592019-03-08 12:39:11 -08003054 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003055 "-f",
3056 "--only-failed",
3057 action="store_true",
3058 help="Run only those tests that failed the previous sanity check "
3059 "invocation.")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003060
Anas Nashif3ba1d432017-12-05 15:28:44 -05003061 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003062 "--retry-failed", type=int, default=0,
3063 help="Retry failing tests again, up to the number of times specified.")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003064
Marc Herbert0c465bb2019-03-11 17:28:36 -07003065 test_xor_subtest = case_select.add_mutually_exclusive_group()
3066
3067 test_xor_subtest.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003068 "-s", "--test", action="append",
3069 help="Run only the specified test cases. These are named by "
Marc Herberte5cedca2019-04-08 14:02:34 -07003070 "<path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003071
Marc Herbert0c465bb2019-03-11 17:28:36 -07003072 test_xor_subtest.add_argument(
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003073 "--sub-test", action="append",
Marc Herbert932a33a2019-03-12 11:37:53 -07003074 help="""Recursively find sub-test functions and run the entire
3075 test section where they were found, including all sibling test
3076 functions. Sub-tests are named by:
3077 section.name.in.testcase.yaml.function_name_without_test_prefix
3078 Example: kernel.fifo.poll.fifo_loop
3079 """)
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003080
Anas Nashif3ba1d432017-12-05 15:28:44 -05003081 parser.add_argument(
3082 "-l", "--all", action="store_true",
3083 help="Build/test on all platforms. Any --platform arguments "
3084 "ignored.")
Andrew Boie6acbe632015-07-17 12:03:52 -07003085
Anas Nashif3ba1d432017-12-05 15:28:44 -05003086 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003087 "-o", "--report-dir",
3088 help="""Output reports containing results of the test run into the
3089 specified directory.
3090 The output will be both in CSV and JUNIT format
3091 (sanitycheck.csv and sanitycheck.xml).
3092 """)
3093
Anas Nashif3ba1d432017-12-05 15:28:44 -05003094 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003095 "--report-name",
3096 help="""Create a report with a custom name.
3097 """)
3098
3099 parser.add_argument("--detailed-report",
3100 action="store",
3101 metavar="FILENAME",
3102 help="""Generate a junit report with detailed testcase results.
3103 Unlike the CSV file produced by --testcase-report, this XML
3104 report includes only tests which have run and none which were
3105 merely built. If an image with multiple tests crashes early then
3106 later tests are not accounted for either.""")
3107
3108 parser.add_argument("--report-excluded",
3109 action="store_true",
3110 help="""List all tests that are never run based on current scope and
3111 coverage. If you are looking for accurate results, run this with
3112 --all, but this will take a while...""")
3113
Daniel Leung7f850102016-04-08 11:07:32 -07003114 parser.add_argument("--compare-report",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003115 help="Use this report file for size comparison")
Daniel Leung7f850102016-04-08 11:07:32 -07003116
Anas Nashif3ba1d432017-12-05 15:28:44 -05003117 parser.add_argument(
3118 "-B", "--subset",
3119 help="Only run a subset of the tests, 1/4 for running the first 25%%, "
3120 "3/5 means run the 3rd fifth of the total. "
3121 "This option is useful when running a large number of tests on "
3122 "different hosts to speed up execution time.")
Anas Nashifa8a13882017-12-30 13:01:06 -05003123
3124 parser.add_argument(
3125 "-N", "--ninja", action="store_true",
Anas Nashif25f6ab62018-03-06 07:15:11 -06003126 help="Use the Ninja generator with CMake")
Anas Nashifa8a13882017-12-30 13:01:06 -05003127
Anas Nashif3ba1d432017-12-05 15:28:44 -05003128 parser.add_argument(
3129 "-y", "--dry-run", action="store_true",
3130 help="Create the filtered list of test cases, but don't actually "
3131 "run them. Useful if you're just interested in "
3132 "--discard-report")
Andrew Boie6acbe632015-07-17 12:03:52 -07003133
Anas Nashif75547e22018-02-24 08:32:14 -06003134 parser.add_argument("--list-tags", action="store_true",
3135 help="list all tags in selected tests")
3136
Marc Herbertedf17592019-03-08 12:39:11 -08003137 case_select.add_argument("--list-tests", action="store_true",
Marc Herbert932a33a2019-03-12 11:37:53 -07003138 help="""List of all sub-test functions recursively found in
3139 all --testcase-root arguments. Note different sub-tests can share
3140 the same section name and come from different directories.
3141 The output is flattened and reports --sub-test names only,
3142 not their directories. For instance net.socket.getaddrinfo_ok
3143 and net.socket.fd_set belong to different directories.
3144 """)
Anas Nashifc0149cc2018-04-14 23:12:58 -05003145
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003146 parser.add_argument("--export-tests", action="store",
3147 metavar="FILENAME",
3148 help="Export tests case meta-data to a file in CSV format.")
3149
Anas Nashife0a6a0b2018-02-15 20:07:24 -06003150
Anas Nashif654ec5982019-04-11 08:38:21 -04003151 parser.add_argument("--timestamps",
3152 action="store_true",
3153 help="Print all messages with time stamps")
3154
Anas Nashif3ba1d432017-12-05 15:28:44 -05003155 parser.add_argument(
3156 "-r", "--release", action="store_true",
3157 help="Update the benchmark database with the results of this test "
3158 "run. Intended to be run by CI when tagging an official "
3159 "release. This database is used as a basis for comparison "
3160 "when looking for deltas in metrics such as footprint")
Andrew Boie6acbe632015-07-17 12:03:52 -07003161 parser.add_argument("-w", "--warnings-as-errors", action="store_true",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003162 help="Treat warning conditions as errors")
3163 parser.add_argument(
3164 "-v",
3165 "--verbose",
3166 action="count",
3167 default=0,
3168 help="Emit debugging information, call multiple times to increase "
3169 "verbosity")
3170 parser.add_argument(
3171 "-i", "--inline-logs", action="store_true",
3172 help="Upon test failure, print relevant log data to stdout "
3173 "instead of just a path to it")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -08003174 parser.add_argument("--log-file", metavar="FILENAME", action="store",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003175 help="log also to file")
3176 parser.add_argument(
3177 "-m", "--last-metrics", action="store_true",
3178 help="Instead of comparing metrics from the last --release, "
3179 "compare with the results of the previous sanity check "
3180 "invocation")
3181 parser.add_argument(
3182 "-u",
3183 "--no-update",
3184 action="store_true",
3185 help="do not update the results of the last run of the sanity "
3186 "checks")
Marc Herbertedf17592019-03-08 12:39:11 -08003187 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003188 "-F",
3189 "--load-tests",
3190 metavar="FILENAME",
3191 action="store",
Marc Herbert932a33a2019-03-12 11:37:53 -07003192 help="Load list of tests and platforms to be run from file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04003193
Marc Herbertedf17592019-03-08 12:39:11 -08003194 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003195 "-E",
3196 "--save-tests",
3197 metavar="FILENAME",
3198 action="store",
Marc Herbert682961a2019-03-20 16:48:49 -07003199 help="Append list of tests and platforms to be run to file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04003200
Andy Doancbecadd2019-02-08 10:19:10 -06003201 test_or_build = parser.add_mutually_exclusive_group()
3202 test_or_build.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003203 "-b", "--build-only", action="store_true",
3204 help="Only build the code, do not execute any of it in QEMU")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003205
Andy Doancbecadd2019-02-08 10:19:10 -06003206 test_or_build.add_argument(
3207 "--test-only", action="store_true",
3208 help="""Only run device tests with current artifacts, do not build
3209 the code""")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003210 parser.add_argument(
Kumar Gala84cf9dc2019-06-15 09:36:06 -05003211 "--cmake-only", action="store_true",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003212 help="Only run cmake, do not build or run.")
Kumar Gala84cf9dc2019-06-15 09:36:06 -05003213
3214 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003215 "-j", "--jobs", type=int,
Marc Herbert9e573382019-07-03 12:49:42 -07003216 help="Number of jobs for building, defaults to number of CPU threads, "
3217 "overcommited by factor 2 when --build-only")
Anas Nashif73440ea2018-02-19 10:57:03 -06003218
3219 parser.add_argument(
Anas Nashif424a3db2018-02-20 08:37:24 -06003220 "--show-footprint", action="store_true",
3221 help="Show footprint statistics and deltas since last release."
3222 )
3223 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003224 "-H", "--footprint-threshold", type=float, default=5,
3225 help="When checking test case footprint sizes, warn the user if "
3226 "the new app size is greater then the specified percentage "
3227 "from the last release. Default is 5. 0 to warn on any "
3228 "increase on app size")
3229 parser.add_argument(
3230 "-D", "--all-deltas", action="store_true",
3231 help="Show all footprint deltas, positive or negative. Implies "
3232 "--footprint-threshold=0")
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003233 parser.add_argument(
3234 "-O", "--outdir",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003235 default=os.path.join(os.getcwd(),"sanity-out"),
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003236 help="Output directory for logs and binaries. "
Anas Nashiff114a132018-11-20 11:51:34 -05003237 "Default is 'sanity-out' in the current directory. "
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003238 "This directory will be deleted unless '--no-clean' is set.")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003239 parser.add_argument(
3240 "-n", "--no-clean", action="store_true",
3241 help="Do not delete the outdir before building. Will result in "
3242 "faster compilation since builds will be incremental")
Marc Herbertedf17592019-03-08 12:39:11 -08003243 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003244 "-T", "--testcase-root", action="append", default=[],
3245 help="Base directory to recursively search for test cases. All "
3246 "testcase.yaml files under here will be processed. May be "
Marc Herbert932a33a2019-03-12 11:37:53 -07003247 "called multiple times. Defaults to the 'samples/' and "
3248 "'tests/' directories at the base of the Zephyr tree.")
Anas Nashif7dd19ea2018-11-14 08:46:49 -05003249
Anas Nashif3ba1d432017-12-05 15:28:44 -05003250 board_root_list = ["%s/boards" % ZEPHYR_BASE,
3251 "%s/scripts/sanity_chk/boards" % ZEPHYR_BASE]
Anas Nashif7dd19ea2018-11-14 08:46:49 -05003252
Anas Nashif3ba1d432017-12-05 15:28:44 -05003253 parser.add_argument(
Thomas Stilwell53105102019-10-23 14:25:09 +02003254 "-A", "--board-root", action="append", default=board_root_list,
Anas Nashif83fc06a2019-06-22 11:04:10 -04003255 help="""Directory to search for board configuration files. All .yaml
3256files in the directory will be processed. The directory should have the same
3257structure in the main Zephyr tree: boards/<arch>/<board_name>/""")
3258
Anas Nashif3ba1d432017-12-05 15:28:44 -05003259 parser.add_argument(
3260 "-z", "--size", action="append",
3261 help="Don't run sanity checks. Instead, produce a report to "
3262 "stdout detailing RAM/ROM sizes on the specified filenames. "
3263 "All other command line arguments ignored.")
3264 parser.add_argument(
3265 "-S", "--enable-slow", action="store_true",
3266 help="Execute time-consuming test cases that have been marked "
3267 "as 'slow' in testcase.yaml. Normally these are only built.")
Sebastian Bøe03ed0952018-11-13 13:36:19 +01003268 parser.add_argument(
3269 "--disable-unrecognized-section-test", action="store_true",
3270 default=False,
3271 help="Skip the 'unrecognized section' test.")
Andrew Boie55121052016-07-20 11:52:04 -07003272 parser.add_argument("-R", "--enable-asserts", action="store_true",
Kumar Gala0d48cbe2018-01-26 10:22:20 -06003273 default=True,
Andrew Boie29599f62018-05-24 13:33:09 -07003274 help="deprecated, left for compatibility")
Kumar Gala0d48cbe2018-01-26 10:22:20 -06003275 parser.add_argument("--disable-asserts", action="store_false",
3276 dest="enable_asserts",
Andrew Boie29599f62018-05-24 13:33:09 -07003277 help="deprecated, left for compatibility")
Anas Nashife3febe92016-11-30 14:25:44 -05003278 parser.add_argument("-Q", "--error-on-deprecations", action="store_false",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003279 help="Error on deprecation warnings.")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003280 parser.add_argument("--enable-size-report", action="store_true",
3281 help="Enable expensive computation of RAM/ROM segment sizes.")
Sebastian Bøec2182612017-11-09 12:25:02 +01003282
3283 parser.add_argument(
3284 "-x", "--extra-args", action="append", default=[],
Alberto Escolar Piedras25e06362018-02-10 17:40:33 +01003285 help="""Extra CMake cache entries to define when building test cases.
3286 May be called multiple times. The key-value entries will be
Sebastian Bøec2182612017-11-09 12:25:02 +01003287 prefixed with -D before being passed to CMake.
3288
3289 E.g
3290 "sanitycheck -x=USE_CCACHE=0"
3291 will translate to
3292 "cmake -DUSE_CCACHE=0"
3293
3294 which will ultimately disable ccache.
3295 """
3296 )
Michael Scott421ce462019-06-18 09:37:46 -07003297
Andy Doan79c48842019-02-08 10:09:04 -06003298 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003299 "--device-testing", action="store_true",
3300 help="Test on device directly. Specify the serial device to "
3301 "use with the --device-serial option.")
3302
3303 parser.add_argument(
3304 "-X", "--fixture", action="append", default=[],
3305 help="Specify a fixture that a board might support")
3306 parser.add_argument(
3307 "--device-serial",
3308 help="Serial device for accessing the board (e.g., /dev/ttyACM0)")
3309
3310 parser.add_argument("--generate-hardware-map",
3311 help="""Probe serial devices connected to this platform
3312 and create a hardware map file to be used with
3313 --device-testing
3314 """)
3315
3316 parser.add_argument("--hardware-map",
3317 help="""Load hardware map from a file. This will be used
3318 for testing on hardware that is listed in the file.
3319 """)
3320
3321 parser.add_argument(
Andy Doan79c48842019-02-08 10:09:04 -06003322 "--west-flash", nargs='?', const=[],
3323 help="""Uses west instead of ninja or make to flash when running with
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +02003324 --device-testing. Supports comma-separated argument list.
Andy Doan79c48842019-02-08 10:09:04 -06003325
Michael Scott4ca54392019-07-09 14:21:30 -07003326 E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +02003327 --west-flash="--board-id=foobar,--erase"
3328 will translate to "west flash -- --board-id=foobar --erase"
Michael Scott4ca54392019-07-09 14:21:30 -07003329
3330 NOTE: device-testing must be enabled to use this option.
Andy Doan79c48842019-02-08 10:09:04 -06003331 """
3332 )
Michael Scott421ce462019-06-18 09:37:46 -07003333 parser.add_argument(
3334 "--west-runner",
3335 help="""Uses the specified west runner instead of default when running
3336 with --west-flash.
3337
3338 E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0
3339 --west-flash --west-runner=pyocd"
3340 will translate to "west flash --runner pyocd"
3341
3342 NOTE: west-flash must be enabled to use this option.
3343 """
3344 )
Andrew Boie8047a6f2019-07-02 15:43:29 -07003345
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003346 parser.add_argument("--enable-coverage", action="store_true",
Anas Nashif8d72bb92018-11-07 23:05:42 -05003347 help="Enable code coverage using gcov.")
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003348
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003349 parser.add_argument("-C", "--coverage", action="store_true",
Andrew Boie8047a6f2019-07-02 15:43:29 -07003350 help="Generate coverage reports. Implies "
3351 "--enable_coverage and --enable-slow")
Andrew Boie6acbe632015-07-17 12:03:52 -07003352
Andrew Boie8047a6f2019-07-02 15:43:29 -07003353 parser.add_argument("--coverage-platform", action="append", default=[],
Anas Nashifdbd76492018-11-23 20:24:19 -05003354 help="Plarforms to run coverage reports on. "
Andrew Boie8047a6f2019-07-02 15:43:29 -07003355 "This option may be used multiple times. "
3356 "Default to what was selected with --platform.")
Anas Nashifdbd76492018-11-23 20:24:19 -05003357
Anas Nashif83fc06a2019-06-22 11:04:10 -04003358 parser.add_argument("--gcov-tool", default=None,
3359 help="Path to the gcov tool to use for code coverage "
3360 "reports")
3361
Andrew Boie6acbe632015-07-17 12:03:52 -07003362 return parser.parse_args()
3363
Anas Nashif3ba1d432017-12-05 15:28:44 -05003364
Andrew Boie6acbe632015-07-17 12:03:52 -07003365def log_info(filename):
Mazen NEIFER8f1dd3a2017-02-04 14:32:04 +01003366 filename = os.path.relpath(os.path.realpath(filename))
Anas Nashif83fc06a2019-06-22 11:04:10 -04003367 if options.inline_logs:
Andrew Boie08ce5a52016-02-22 13:28:10 -08003368 info("{:-^100}".format(filename))
Andrew Boied01535c2017-01-10 13:15:02 -08003369
3370 try:
3371 with open(filename) as fp:
3372 data = fp.read()
3373 except Exception as e:
3374 data = "Unable to read log data (%s)\n" % (str(e))
3375
3376 sys.stdout.write(data)
3377 if log_file:
3378 log_file.write(data)
Andrew Boie08ce5a52016-02-22 13:28:10 -08003379 info("{:-^100}".format(filename))
Andrew Boie6acbe632015-07-17 12:03:52 -07003380 else:
Andrew Boie08ce5a52016-02-22 13:28:10 -08003381 info("\tsee: " + COLOR_YELLOW + filename + COLOR_NORMAL)
Andrew Boie6acbe632015-07-17 12:03:52 -07003382
Andrew Boiebbd670c2015-08-17 13:16:11 -07003383def size_report(sc):
3384 info(sc.filename)
Andrew Boie73b4ee62015-10-07 11:33:22 -07003385 info("SECTION NAME VMA LMA SIZE HEX SZ TYPE")
Andrew Boie9882dcd2015-10-07 14:25:51 -07003386 for i in range(len(sc.sections)):
3387 v = sc.sections[i]
3388
Andrew Boie73b4ee62015-10-07 11:33:22 -07003389 info("%-17s 0x%08x 0x%08x %8d 0x%05x %-7s" %
3390 (v["name"], v["virt_addr"], v["load_addr"], v["size"], v["size"],
3391 v["type"]))
Andrew Boie9882dcd2015-10-07 14:25:51 -07003392
Andrew Boie73b4ee62015-10-07 11:33:22 -07003393 info("Totals: %d bytes (ROM), %d bytes (RAM)" %
Anas Nashif3ba1d432017-12-05 15:28:44 -05003394 (sc.rom_size, sc.ram_size))
Andrew Boiebbd670c2015-08-17 13:16:11 -07003395 info("")
3396
Anas Nashiff29087e2019-01-25 09:37:38 -05003397def retrieve_gcov_data(intput_file):
Anas Nashifdb9592a2018-10-08 10:19:41 -04003398 if VERBOSE:
3399 print("Working on %s" %intput_file)
3400 extracted_coverage_info = {}
3401 capture_data = False
Anas Nashiff29087e2019-01-25 09:37:38 -05003402 capture_complete = False
Anas Nashifdb9592a2018-10-08 10:19:41 -04003403 with open(intput_file, 'r') as fp:
3404 for line in fp.readlines():
3405 if re.search("GCOV_COVERAGE_DUMP_START", line):
3406 capture_data = True
3407 continue
3408 if re.search("GCOV_COVERAGE_DUMP_END", line):
Anas Nashiff29087e2019-01-25 09:37:38 -05003409 capture_complete = True
Anas Nashifdb9592a2018-10-08 10:19:41 -04003410 break
3411 # Loop until the coverage data is found.
3412 if not capture_data:
3413 continue
3414 if line.startswith("*"):
3415 sp = line.split("<")
3416 if len(sp) > 1:
3417 # Remove the leading delimiter "*"
3418 file_name = sp[0][1:]
3419 # Remove the trailing new line char
3420 hex_dump = sp[1][:-1]
3421 else:
3422 continue
3423 else:
3424 continue
3425 extracted_coverage_info.update({file_name:hex_dump})
Anas Nashiff29087e2019-01-25 09:37:38 -05003426 if not capture_data:
3427 capture_complete = True
3428 return {'complete': capture_complete, 'data': extracted_coverage_info}
Anas Nashifdb9592a2018-10-08 10:19:41 -04003429
3430def create_gcda_files(extracted_coverage_info):
3431 if VERBOSE:
3432 print("Generating gcda files")
3433 for filename, hexdump_val in extracted_coverage_info.items():
3434 # if kobject_hash is given for coverage gcovr fails
3435 # hence skipping it problem only in gcovr v4.1
3436 if "kobject_hash" in filename:
3437 filename = (filename[:-4]) +"gcno"
3438 try:
3439 os.remove(filename)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003440 except Exception:
Anas Nashifdb9592a2018-10-08 10:19:41 -04003441 pass
3442 continue
3443
3444 with open(filename, 'wb') as fp:
3445 fp.write(bytes.fromhex(hexdump_val))
Anas Nashif3ba1d432017-12-05 15:28:44 -05003446
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003447def generate_coverage(outdir, ignores):
Anas Nashifdb9592a2018-10-08 10:19:41 -04003448
3449 for filename in glob.glob("%s/**/handler.log" %outdir, recursive=True):
Anas Nashiff29087e2019-01-25 09:37:38 -05003450 gcov_data = retrieve_gcov_data(filename)
3451 capture_complete = gcov_data['complete']
3452 extracted_coverage_info = gcov_data['data']
3453 if capture_complete:
3454 create_gcda_files(extracted_coverage_info)
3455 verbose("Gcov data captured: {}".format(filename))
3456 else:
3457 error("Gcov data capture incomplete: {}".format(filename))
Anas Nashifdb9592a2018-10-08 10:19:41 -04003458
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003459 gcov_tool = options.gcov_tool
3460
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003461 with open(os.path.join(outdir, "coverage.log"), "a") as coveragelog:
3462 coveragefile = os.path.join(outdir, "coverage.info")
3463 ztestfile = os.path.join(outdir, "ztest.info")
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003464 subprocess.call(["lcov", "--gcov-tool", gcov_tool,
3465 "--capture", "--directory", outdir,
3466 "--rc", "lcov_branch_coverage=1",
3467 "--output-file", coveragefile], stdout=coveragelog)
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003468 # We want to remove tests/* and tests/ztest/test/* but save tests/ztest
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003469 subprocess.call(["lcov", "--gcov-tool", gcov_tool, "--extract", coveragefile,
Anas Nashif3ba1d432017-12-05 15:28:44 -05003470 os.path.join(ZEPHYR_BASE, "tests", "ztest", "*"),
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003471 "--output-file", ztestfile,
3472 "--rc", "lcov_branch_coverage=1"], stdout=coveragelog)
3473
Anas Nashif3cbffef2018-11-07 23:50:54 -05003474 if os.path.exists(ztestfile) and os.path.getsize(ztestfile) > 0:
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003475 subprocess.call(["lcov", "--gcov-tool", gcov_tool, "--remove", ztestfile,
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003476 os.path.join(ZEPHYR_BASE, "tests/ztest/test/*"),
3477 "--output-file", ztestfile,
3478 "--rc", "lcov_branch_coverage=1"],
3479 stdout=coveragelog)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003480 files = [coveragefile, ztestfile]
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003481 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003482 files = [coveragefile]
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003483
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003484 for i in ignores:
Anas Nashif3ba1d432017-12-05 15:28:44 -05003485 subprocess.call(
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003486 ["lcov", "--gcov-tool", gcov_tool, "--remove",
3487 coveragefile, i, "--output-file",
3488 coveragefile, "--rc", "lcov_branch_coverage=1"],
Anas Nashif3ba1d432017-12-05 15:28:44 -05003489 stdout=coveragelog)
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003490
Alberto Escolar Piedras834b86f2019-02-03 15:48:07 +01003491 #The --ignore-errors source option is added to avoid it exiting due to
3492 #samples/application_development/external_lib/
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003493 ret = subprocess.call(["genhtml", "--legend", "--branch-coverage",
Alberto Escolar Piedras834b86f2019-02-03 15:48:07 +01003494 "--ignore-errors", "source",
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003495 "-output-directory",
3496 os.path.join(outdir, "coverage")] + files,
3497 stdout=coveragelog)
3498 if ret==0:
3499 info("HTML report generated: %s"%
Anas Nashif83fc06a2019-06-22 11:04:10 -04003500 os.path.join(outdir, "coverage","index.html"))
Anas Nashif3ba1d432017-12-05 15:28:44 -05003501
Anas Nashif83fc06a2019-06-22 11:04:10 -04003502def get_generator():
3503 if options.ninja:
3504 generator_cmd = "ninja"
3505 generator = "Ninja"
3506 else:
3507 generator_cmd = "make"
3508 generator = "Unix Makefiles"
3509 return generator_cmd, generator
3510
3511
3512def export_tests(filename, tests):
3513 with open(filename, "wt") as csvfile:
3514 fieldnames = ['section', 'subsection', 'title', 'reference']
3515 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
3516 for test in tests:
3517 data = test.split(".")
3518 subsec = " ".join(data[1].split("_")).title()
3519 rowdict = {
3520 "section": data[0].capitalize(),
3521 "subsection": subsec,
3522 "title": test,
3523 "reference": test
3524 }
3525 cw.writerow(rowdict)
3526
3527def get_unique_tests(suite):
3528 unq = []
3529 run_individual_tests = []
3530 for _, tc in suite.testcases.items():
3531 for c in tc.cases:
3532 if options.sub_test and c in options.sub_test:
3533 if tc.name not in run_individual_tests:
3534 run_individual_tests.append(tc.name)
3535 unq.append(c)
3536
3537 return unq
3538
3539
3540
3541def native_and_unit_first(a, b):
3542 if a[0].startswith('unit_testing'):
3543 return -1
3544 if b[0].startswith('unit_testing'):
3545 return 1
3546 if a[0].startswith('native_posix'):
3547 return -1
3548 if b[0].startswith('native_posix'):
3549 return 1
3550 if a[0].split("/",1)[0].endswith("_bsim"):
3551 return -1
3552 if b[0].split("/",1)[0].endswith("_bsim"):
3553 return 1
3554
3555 return (a > b) - (a < b)
3556
3557
3558run_individual_tests = None
3559options = None
Andrew Boiebbd670c2015-08-17 13:16:11 -07003560
Andrew Boie6acbe632015-07-17 12:03:52 -07003561def main():
Andrew Boie4b182472015-07-31 12:25:22 -07003562 start_time = time.time()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003563 global VERBOSE, log_file
Anas Nashife10b6512017-12-30 13:01:45 -05003564 global options
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003565 global run_individual_tests
Andrew Boie1578ef72019-07-03 10:19:29 -07003566
Anas Nashife10b6512017-12-30 13:01:45 -05003567 options = parse_arguments()
Andrew Boiebbd670c2015-08-17 13:16:11 -07003568
Anas Nashif83fc06a2019-06-22 11:04:10 -04003569
3570 if options.generate_hardware_map:
3571 from serial.tools import list_ports
3572 serial_devices = list_ports.comports()
3573 filtered = []
3574 for d in serial_devices:
3575 if d.manufacturer in ['ARM', 'SEGGER', 'MBED', 'STMicroelectronics', 'Atmel Corp.']:
3576 s_dev = {}
3577 s_dev['platform'] = "unknown"
3578 s_dev['id'] = d.serial_number
3579 s_dev['serial'] = d.device
3580 s_dev['product'] = d.product
3581 if s_dev['product'] in ['DAPLink CMSIS-DAP', 'MBED CMSIS-DAP']:
3582 s_dev['runner'] = "pyocd"
3583 else:
3584 s_dev['runner'] = "unknown"
3585 s_dev['available'] = True
3586 s_dev['connected'] = True
3587 filtered.append(s_dev)
3588 else:
3589 print("Unsupported device (%s): %s" %(d.manufacturer, d))
3590
3591 if os.path.exists(options.generate_hardware_map):
3592 # use existing map
3593
3594 with open(options.generate_hardware_map, 'r') as yaml_file:
3595 hwm = yaml.load(yaml_file, Loader=yaml.FullLoader)
3596 # disconnect everything
3597 for h in hwm:
3598 h['connected'] = False
3599
3600 for d in filtered:
3601 for h in hwm:
3602 if d['id'] == h['id'] and d['product'] == h['product']:
3603 print("Already in map: %s (%s)" %(d['product'], d['id']))
3604 h['connected'] = True
3605 h['serial'] = d['serial']
3606 d['match'] = True
3607
3608 new = list(filter(lambda n: not n.get('match', False), filtered))
3609 hwm = hwm + new
3610
3611 #import pprint
3612 #pprint.pprint(hwm)
3613 with open(options.generate_hardware_map, 'w') as yaml_file:
3614 yaml.dump(hwm, yaml_file, default_flow_style=False)
3615
3616
3617 else:
3618 # create new file
3619 with open(options.generate_hardware_map, 'w') as yaml_file:
3620 yaml.dump(filtered, yaml_file, default_flow_style=False)
3621
3622 return
3623
3624
Michael Scott421ce462019-06-18 09:37:46 -07003625 if options.west_runner and not options.west_flash:
3626 error("west-runner requires west-flash to be enabled")
3627 sys.exit(1)
3628
Michael Scott4ca54392019-07-09 14:21:30 -07003629 if options.west_flash and not options.device_testing:
3630 error("west-flash requires device-testing to be enabled")
3631 sys.exit(1)
3632
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003633 if options.coverage:
3634 options.enable_coverage = True
Andrew Boie8047a6f2019-07-02 15:43:29 -07003635 options.enable_slow = True
3636 if not options.coverage_platform:
3637 options.coverage_platform = options.platform
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003638
Anas Nashife10b6512017-12-30 13:01:45 -05003639 if options.size:
3640 for fn in options.size:
Andrew Boie52fef672016-11-29 12:21:59 -08003641 size_report(SizeCalculator(fn, []))
Andrew Boiebbd670c2015-08-17 13:16:11 -07003642 sys.exit(0)
3643
Anas Nashife10b6512017-12-30 13:01:45 -05003644 VERBOSE += options.verbose
Anas Nashif83fc06a2019-06-22 11:04:10 -04003645
Anas Nashife10b6512017-12-30 13:01:45 -05003646 if options.log_file:
3647 log_file = open(options.log_file, "w")
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03003648
Anas Nashife10b6512017-12-30 13:01:45 -05003649 if options.subset:
3650 subset, sets = options.subset.split("/")
Anas Nashif035799f2017-05-13 21:31:53 -04003651 if int(subset) > 0 and int(sets) >= int(subset):
Anas Nashif3ba1d432017-12-05 15:28:44 -05003652 info("Running only a subset: %s/%s" % (subset, sets))
Anas Nashif035799f2017-05-13 21:31:53 -04003653 else:
Anas Nashife10b6512017-12-30 13:01:45 -05003654 error("You have provided a wrong subset value: %s." % options.subset)
Anas Nashif035799f2017-05-13 21:31:53 -04003655 return
3656
Anas Nashif83fc06a2019-06-22 11:04:10 -04003657 # Cleanup
3658
3659 if options.no_clean or options.only_failed or options.test_only:
3660 if os.path.exists(options.outdir):
3661 info("Keeping artifacts untouched")
3662 elif os.path.exists(options.outdir):
3663 for i in range(1,100):
3664 new_out = options.outdir + ".{}".format(i)
3665 if not os.path.exists(new_out):
3666 info("Renaming output directory to {}".format(new_out))
3667 shutil.move(options.outdir, new_out)
3668 break
3669 #shutil.rmtree("%s.old" %options.outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07003670
Anas Nashife10b6512017-12-30 13:01:45 -05003671 if not options.testcase_root:
3672 options.testcase_root = [os.path.join(ZEPHYR_BASE, "tests"),
Andrew Boie3d348712016-04-08 11:52:13 -07003673 os.path.join(ZEPHYR_BASE, "samples")]
3674
Anas Nashif83fc06a2019-06-22 11:04:10 -04003675 suite = TestSuite(options.board_root, options.testcase_root, options.outdir)
3676 suite.add_testcases()
3677 suite.add_configurations()
Anas Nashifbd166f42017-09-02 12:32:08 -04003678
Anas Nashif83fc06a2019-06-22 11:04:10 -04003679 if options.device_testing:
3680 if options.hardware_map:
3681 suite.load_hardware_map(options.hardware_map)
3682 if not options.platform:
3683 options.platform = []
3684 for platform in suite.connected_hardware:
3685 if platform['connected']:
3686 options.platform.append(platform['platform'])
3687
3688 elif options.device_serial: #back-ward compatibility
3689 if options.platform and len(options.platform) == 1:
3690 suite.load_hardware_map_from_cmdline(options.device_serial,
3691 options.platform[0])
3692 else:
3693 error("""When --device-testing is used with --device-serial, only one
3694 platform is allowed""")
3695
3696
3697
3698 if suite.load_errors:
Kumar Galac84235e2018-04-10 13:32:51 -05003699 sys.exit(1)
3700
Anas Nashif75547e22018-02-24 08:32:14 -06003701 if options.list_tags:
3702 tags = set()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003703 for _, tc in suite.testcases.items():
Anas Nashif75547e22018-02-24 08:32:14 -06003704 tags = tags.union(tc.tags)
3705
3706 for t in tags:
3707 print("- {}".format(t))
3708
3709 return
3710
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003711 if options.export_tests:
3712 cnt = 0
3713 unq = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04003714 for _, tc in suite.testcases.items():
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003715 for c in tc.cases:
3716 unq.append(c)
3717
3718 tests = sorted(set(unq))
3719 export_tests(options.export_tests, tests)
3720 return
3721
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003722 run_individual_tests = []
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003723
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003724 if options.test:
3725 run_individual_tests = options.test
3726
Anas Nashif49b22d42019-06-14 13:45:34 -04003727 if options.list_tests or options.sub_test:
3728 cnt = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04003729 unq = get_unique_tests(suite)
Anas Nashif49b22d42019-06-14 13:45:34 -04003730
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003731 if options.sub_test:
3732 if run_individual_tests:
3733 info("Running the following tests:")
3734 for t in run_individual_tests:
3735 print(" - {}".format(t))
3736 else:
3737 info("Tests not found")
3738 return
3739
3740 elif options.list_tests:
3741 for u in sorted(set(unq)):
3742 cnt = cnt + 1
3743 print(" - {}".format(u))
3744 print("{} total.".format(cnt))
3745 return
Anas Nashifc0149cc2018-04-14 23:12:58 -05003746
Anas Nashifbd166f42017-09-02 12:32:08 -04003747 discards = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04003748
3749 if options.only_failed:
3750 suite.get_last_failed()
3751 elif options.load_tests:
3752 suite.load_from_file(options.load_tests)
3753 elif options.test_only:
3754 last_run = os.path.join(options.outdir, "sanitycheck.csv")
3755 suite.load_from_file(last_run)
Anas Nashifbd166f42017-09-02 12:32:08 -04003756 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003757 discards = suite.apply_filters()
Andrew Boie6acbe632015-07-17 12:03:52 -07003758
Anas Nashif30551f42018-01-12 21:56:59 -05003759 if VERBOSE > 1 and discards:
Anas Nashiff18e2ab2018-01-13 07:57:42 -05003760 # if we are using command line platform filter, no need to list every
3761 # other platform as excluded, we know that already.
3762 # Show only the discards that apply to the selected platforms on the
3763 # command line
3764
Andrew Boie08ce5a52016-02-22 13:28:10 -08003765 for i, reason in discards.items():
Anas Nashiff18e2ab2018-01-13 07:57:42 -05003766 if options.platform and i.platform.name not in options.platform:
3767 continue
Anas Nashif3ba1d432017-12-05 15:28:44 -05003768 debug(
3769 "{:<25} {:<50} {}SKIPPED{}: {}".format(
3770 i.platform.name,
Anas Nashif83fc06a2019-06-22 11:04:10 -04003771 i.testcase.name,
Anas Nashif3ba1d432017-12-05 15:28:44 -05003772 COLOR_YELLOW,
3773 COLOR_NORMAL,
3774 reason))
Andrew Boie6acbe632015-07-17 12:03:52 -07003775
Anas Nashif49b22d42019-06-14 13:45:34 -04003776 if options.report_excluded:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003777 all_tests = set(get_unique_tests(suite))
Anas Nashif49b22d42019-06-14 13:45:34 -04003778 to_be_run = set()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003779 for i,p in suite.instances.items():
3780 to_be_run.update(p.testcase.cases)
Anas Nashif49b22d42019-06-14 13:45:34 -04003781
Anas Nashif83fc06a2019-06-22 11:04:10 -04003782 if all_tests - to_be_run:
Anas Nashif49b22d42019-06-14 13:45:34 -04003783 print("Tests that never build or run:")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003784 for not_run in all_tests - to_be_run:
Anas Nashif49b22d42019-06-14 13:45:34 -04003785 print("- {}".format(not_run))
3786
3787 return
3788
Anas Nashife10b6512017-12-30 13:01:45 -05003789 if options.subset:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003790 #suite.instances = OrderedDict(sorted(suite.instances.items(),
3791 # key=cmp_to_key(native_and_unit_first)))
Anas Nashife10b6512017-12-30 13:01:45 -05003792 subset, sets = options.subset.split("/")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003793 total = len(suite.instances)
Anas Nashif035799f2017-05-13 21:31:53 -04003794 per_set = round(total / int(sets))
Anas Nashif3ba1d432017-12-05 15:28:44 -05003795 start = (int(subset) - 1) * per_set
Anas Nashif035799f2017-05-13 21:31:53 -04003796 if subset == sets:
3797 end = total
3798 else:
3799 end = start + per_set
3800
Anas Nashif83fc06a2019-06-22 11:04:10 -04003801 sliced_instances = islice(suite.instances.items(), start, end)
3802 suite.instances = OrderedDict(sliced_instances)
Anas Nashif035799f2017-05-13 21:31:53 -04003803
Andrew Boie6acbe632015-07-17 12:03:52 -07003804
Anas Nashif83fc06a2019-06-22 11:04:10 -04003805 if options.save_tests:
3806 suite.csv_report(options.save_tests)
Andrew Boie6acbe632015-07-17 12:03:52 -07003807 return
3808
Anas Nashif83fc06a2019-06-22 11:04:10 -04003809 info("%d test configurations selected, %d configurations discarded due to filters." %
3810 (len(suite.instances), len(discards)))
3811
3812 if options.dry_run:
3813 duration = time.time() - start_time
3814 info("Completed in %d seconds" % (duration))
3815 return
3816
3817 retries = options.retry_failed + 1
3818 completed = 0
3819
3820 suite.update()
3821 suite.start_time = start_time
3822
3823 while True:
3824 completed += 1
3825
3826 if completed > 1:
3827 info("%d Iteration:" %(completed ))
3828 time.sleep(60) # waiting for the system to settle down
3829 suite.total_done = suite.total_tests - suite.total_failed
3830 suite.total_failed = 0
3831
3832 suite.execute()
Anas Nashif654ec5982019-04-11 08:38:21 -04003833 info("", False)
Andrew Boie6acbe632015-07-17 12:03:52 -07003834
Anas Nashif83fc06a2019-06-22 11:04:10 -04003835 retries = retries - 1
3836 if retries == 0 or suite.total_failed == 0:
3837 break
Anas Nashife0a6a0b2018-02-15 20:07:24 -06003838
Anas Nashif83fc06a2019-06-22 11:04:10 -04003839 suite.misc_reports(options.compare_report, options.show_footprint,
3840 options.all_deltas, options.footprint_threshold, options.last_metrics)
3841 suite.save_reports()
Andrew Boie6acbe632015-07-17 12:03:52 -07003842
Anas Nashife10b6512017-12-30 13:01:45 -05003843 if options.coverage:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003844 if options.gcov_tool is None:
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02003845 use_system_gcov = False
Andrew Boie49cf4862019-07-08 12:02:13 -07003846
3847 for plat in options.coverage_platform:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003848 ts_plat = suite.get_platform(plat)
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02003849 if ts_plat and (ts_plat.type in {"native", "unit"}):
3850 use_system_gcov = True
Andrew Boie49cf4862019-07-08 12:02:13 -07003851
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02003852 if use_system_gcov or "ZEPHYR_SDK_INSTALL_DIR" not in os.environ:
Andrew Boie49cf4862019-07-08 12:02:13 -07003853 options.gcov_tool = "gcov"
3854 else:
3855 options.gcov_tool = os.path.join(os.environ["ZEPHYR_SDK_INSTALL_DIR"],
3856 "i586-zephyr-elf/bin/i586-zephyr-elf-gcov")
3857
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003858 info("Generating coverage files...")
Anas Nashif4df6c562018-11-07 19:29:04 -05003859 generate_coverage(options.outdir, ["*generated*", "tests/*", "samples/*"])
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003860
Anas Nashif83fc06a2019-06-22 11:04:10 -04003861 suite.duration = time.time() - start_time
3862 suite.summary(options.disable_unrecognized_section_test)
Andrew Boie6acbe632015-07-17 12:03:52 -07003863
Anas Nashif83fc06a2019-06-22 11:04:10 -04003864 if options.device_testing:
3865 print("\nHardware distribution summary:\n")
3866 for p in suite.connected_hardware:
3867 if p['connected']:
3868 print("%s (%s): %d" %(p['platform'], p.get('id', None), p['counter']))
3869
3870 if suite.total_failed or (suite.warnings and options.warnings_as_errors):
Andrew Boie6acbe632015-07-17 12:03:52 -07003871 sys.exit(1)
3872
Andrew Boie6acbe632015-07-17 12:03:52 -07003873if __name__ == "__main__":
3874 main()