blob: 96cfdef8d68807f0fd5d3221f014c7b6775ae635 [file] [log] [blame]
Andrew Boie08ce5a52016-02-22 13:28:10 -08001#!/usr/bin/env python3
Anas Nashifa792a3d2017-04-04 18:47:49 -04002# vim: set syntax=python ts=4 :
Anas Nashif3ae52622019-04-06 09:08:09 -04003# SPDX-License-Identifier: Apache-2.0
Andrew Boie6acbe632015-07-17 12:03:52 -07004"""Zephyr Sanity Tests
5
Marc Herberte5cedca2019-04-08 14:02:34 -07006Also check the "User and Developer Guides" at https://docs.zephyrproject.org/
7
Andrew Boie6acbe632015-07-17 12:03:52 -07008This script scans for the set of unit test applications in the git
9repository and attempts to execute them. By default, it tries to
10build each test case on one platform per architecture, using a precedence
Paul Sokolovskyff70add2017-06-16 01:31:54 +030011list defined in an architecture configuration file, and if possible
Anas Nashif83fc06a2019-06-22 11:04:10 -040012run the tests in any available emulators or simulators on the system.
Andrew Boie6acbe632015-07-17 12:03:52 -070013
Anas Nashifa792a3d2017-04-04 18:47:49 -040014Test cases are detected by the presence of a 'testcase.yaml' or a sample.yaml
15files in the application's project directory. This file may contain one or more
16blocks, each identifying a test scenario. The title of the block is a name for
17the test case, which only needs to be unique for the test cases specified in
18that testcase meta-data. The full canonical name for each test case is <path to
19test case>/<block>.
Andrew Boie6acbe632015-07-17 12:03:52 -070020
Anas Nashif3ba1d432017-12-05 15:28:44 -050021Each test block in the testcase meta data can define the following key/value
22pairs:
Andrew Boie6acbe632015-07-17 12:03:52 -070023
Anas Nashiffa695d22017-10-04 16:14:27 -040024 tags: <list of tags> (required)
Andrew Boie6acbe632015-07-17 12:03:52 -070025 A set of string tags for the testcase. Usually pertains to
26 functional domains but can be anything. Command line invocations
27 of this script can filter the set of tests to run based on tag.
28
Anas Nashifa792a3d2017-04-04 18:47:49 -040029 skip: <True|False> (default False)
Anas Nashif2bd99bc2015-10-12 13:10:57 -040030 skip testcase unconditionally. This can be used for broken tests.
31
Anas Nashifa792a3d2017-04-04 18:47:49 -040032 slow: <True|False> (default False)
Andy Rossdc4151f2019-01-03 14:17:43 -080033 Don't build or run this test case unless --enable-slow was passed
34 in on the command line. Intended for time-consuming test cases
35 that are only run under certain circumstances, like daily
36 builds.
Andrew Boie6bb087c2016-02-10 13:39:00 -080037
Anas Nashifa792a3d2017-04-04 18:47:49 -040038 extra_args: <list of extra arguments>
Sebastian Bøec2182612017-11-09 12:25:02 +010039 Extra cache entries to pass to CMake when building or running the
Andrew Boie6acbe632015-07-17 12:03:52 -070040 test case.
41
Anas Nashifebc329d2017-10-17 09:00:33 -040042 extra_configs: <list of extra configurations>
43 Extra configuration options to be merged with a master prj.conf
44 when building or running the test case.
45
Anas Nashifa792a3d2017-04-04 18:47:49 -040046 build_only: <True|False> (default False)
Marc Herberte5cedca2019-04-08 14:02:34 -070047 If true, don't try to run the test even if the selected platform
48 supports it.
Andrew Boie6acbe632015-07-17 12:03:52 -070049
Anas Nashifa792a3d2017-04-04 18:47:49 -040050 build_on_all: <True|False> (default False)
51 If true, attempt to build test on all available platforms.
52
53 depends_on: <list of features>
54 A board or platform can announce what features it supports, this option
55 will enable the test only those platforms that provide this feature.
56
57 min_ram: <integer>
58 minimum amount of RAM needed for this test to build and run. This is
59 compared with information provided by the board metadata.
60
61 min_flash: <integer>
62 minimum amount of ROM needed for this test to build and run. This is
63 compared with information provided by the board metadata.
64
65 timeout: <number of seconds>
Anas Nashif83fc06a2019-06-22 11:04:10 -040066 Length of time to run test in emulator before automatically killing it.
Andrew Boie6acbe632015-07-17 12:03:52 -070067 Default to 60 seconds.
68
Anas Nashifa792a3d2017-04-04 18:47:49 -040069 arch_whitelist: <list of arches, such as x86, arm, arc>
Andrew Boie6acbe632015-07-17 12:03:52 -070070 Set of architectures that this test case should only be run for.
71
Anas Nashifa792a3d2017-04-04 18:47:49 -040072 arch_exclude: <list of arches, such as x86, arm, arc>
Anas Nashif30d13872015-10-05 10:02:45 -040073 Set of architectures that this test case should not run on.
74
Anas Nashifa792a3d2017-04-04 18:47:49 -040075 platform_whitelist: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040076 Set of platforms that this test case should only be run for.
77
Anas Nashifa792a3d2017-04-04 18:47:49 -040078 platform_exclude: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040079 Set of platforms that this test case should not run on.
Andrew Boie6acbe632015-07-17 12:03:52 -070080
Anas Nashifa792a3d2017-04-04 18:47:49 -040081 extra_sections: <list of extra binary sections>
Andrew Boie52fef672016-11-29 12:21:59 -080082 When computing sizes, sanitycheck will report errors if it finds
83 extra, unexpected sections in the Zephyr binary unless they are named
84 here. They will not be included in the size calculation.
85
Anas Nashifa792a3d2017-04-04 18:47:49 -040086 filter: <expression>
Andrew Boie3ea78922016-03-24 14:46:00 -070087 Filter whether the testcase should be run by evaluating an expression
88 against an environment containing the following values:
89
90 { ARCH : <architecture>,
91 PLATFORM : <platform>,
Javier B Perez79414542016-08-08 12:24:59 -050092 <all CONFIG_* key/value pairs in the test's generated defconfig>,
Anas Nashif45a97862019-01-09 08:46:42 -050093 <all DT_* key/value pairs in the test's generated device tree file>,
94 <all CMake key/value pairs in the test's generated CMakeCache.txt file>,
Javier B Perez79414542016-08-08 12:24:59 -050095 *<env>: any environment variable available
Andrew Boie3ea78922016-03-24 14:46:00 -070096 }
97
98 The grammar for the expression language is as follows:
99
100 expression ::= expression "and" expression
101 | expression "or" expression
102 | "not" expression
103 | "(" expression ")"
104 | symbol "==" constant
105 | symbol "!=" constant
106 | symbol "<" number
107 | symbol ">" number
108 | symbol ">=" number
109 | symbol "<=" number
110 | symbol "in" list
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700111 | symbol ":" string
Andrew Boie3ea78922016-03-24 14:46:00 -0700112 | symbol
113
114 list ::= "[" list_contents "]"
115
116 list_contents ::= constant
117 | list_contents "," constant
118
119 constant ::= number
120 | string
121
122
123 For the case where expression ::= symbol, it evaluates to true
124 if the symbol is defined to a non-empty string.
125
126 Operator precedence, starting from lowest to highest:
127
128 or (left associative)
129 and (left associative)
130 not (right associative)
131 all comparison operators (non-associative)
132
133 arch_whitelist, arch_exclude, platform_whitelist, platform_exclude
134 are all syntactic sugar for these expressions. For instance
135
136 arch_exclude = x86 arc
137
138 Is the same as:
139
140 filter = not ARCH in ["x86", "arc"]
Andrew Boie6acbe632015-07-17 12:03:52 -0700141
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700142 The ':' operator compiles the string argument as a regular expression,
143 and then returns a true value only if the symbol's value in the environment
Anas Nashif578ae402019-07-12 07:54:35 -0700144 matches. For example, if CONFIG_SOC="stm32f107xc" then
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700145
Anas Nashif578ae402019-07-12 07:54:35 -0700146 filter = CONFIG_SOC : "stm.*"
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700147
148 Would match it.
149
Anas Nashifa792a3d2017-04-04 18:47:49 -0400150The set of test cases that actually run depends on directives in the testcase
151filed and options passed in on the command line. If there is any confusion,
Anas Nashif12d8cce2019-11-20 03:47:27 -0800152running with -v or examining the discard report (sanitycheck_discard.csv)
153can help show why particular test cases were skipped.
Andrew Boie6acbe632015-07-17 12:03:52 -0700154
155Metrics (such as pass/fail state and binary size) for the last code
156release are stored in scripts/sanity_chk/sanity_last_release.csv.
157To update this, pass the --all --release options.
158
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -0500159To load arguments from a file, write '+' before the file name, e.g.,
160+file_name. File content must be one or more valid arguments separated by
161line break instead of white spaces.
162
Andrew Boie6acbe632015-07-17 12:03:52 -0700163Most everyday users will run with no arguments.
Andy Rossdc4151f2019-01-03 14:17:43 -0800164
Andrew Boie6acbe632015-07-17 12:03:52 -0700165"""
166
Sebastian Bøe56d74712019-01-21 15:48:46 +0100167import os
Anas Nashifaae71d72018-04-21 22:26:48 -0500168import contextlib
Anas Nashifa4c368e2018-10-15 09:45:59 -0400169import string
Anas Nashifaae71d72018-04-21 22:26:48 -0500170import mmap
Andrew Boie6acbe632015-07-17 12:03:52 -0700171import argparse
Andrew Boie6acbe632015-07-17 12:03:52 -0700172import sys
Andrew Boie6acbe632015-07-17 12:03:52 -0700173import re
Andrew Boie6acbe632015-07-17 12:03:52 -0700174import subprocess
175import multiprocessing
176import select
177import shutil
Marc Herbertaf1090c2019-04-30 14:11:29 -0700178import shlex
Andrew Boie6acbe632015-07-17 12:03:52 -0700179import signal
180import threading
Anas Nashif83fc06a2019-06-22 11:04:10 -0400181import concurrent.futures
182from threading import BoundedSemaphore
183import queue
Andrew Boie6acbe632015-07-17 12:03:52 -0700184import time
Andrew Boie6acbe632015-07-17 12:03:52 -0700185import csv
Anas Nashif83fc06a2019-06-22 11:04:10 -0400186import yaml
Andrew Boie5d4eb782015-10-02 10:04:56 -0700187import glob
Anas Nashif73440ea2018-02-19 10:57:03 -0600188import serial
Daniel Leung6b170072016-04-07 12:10:25 -0700189import concurrent
Anas Nashifb3311ed2017-04-13 14:44:48 -0400190import xml.etree.ElementTree as ET
Anas Nashif7a361b82019-12-06 11:37:40 -0500191import logging
Anas Nashif035799f2017-05-13 21:31:53 -0400192from collections import OrderedDict
193from itertools import islice
Anas Nashife24350c2018-07-11 15:09:22 -0500194from pathlib import Path
195from distutils.spawn import find_executable
Anas Nashifd9882382019-12-12 09:58:28 -0500196
Anas Nashif434995c2019-12-01 13:55:11 -0500197try:
198 from anytree import Node, RenderTree, find
199except ImportError:
200 print("Install the anytree module to use the --test-tree option")
201
Anas Nashif5f908822019-11-25 08:19:25 -0500202try:
203 from tabulate import tabulate
204except ImportError:
205 print("Install tabulate python module with pip to use --device-testing option.")
Andrew Boie6acbe632015-07-17 12:03:52 -0700206
Kumar Gala7733b942019-09-12 17:08:43 -0500207ZEPHYR_BASE = os.getenv("ZEPHYR_BASE")
208if not ZEPHYR_BASE:
209 sys.exit("$ZEPHYR_BASE environment variable undefined")
210
211sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts", "dts"))
212import edtlib
Anas Nashif83fc06a2019-06-22 11:04:10 -0400213
Anas Nashif83fc06a2019-06-22 11:04:10 -0400214hw_map_local = threading.Lock()
Anas Nashifc1ea4522019-10-11 07:32:45 -0700215report_lock = threading.Lock()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400216
Marc Herbert1c8632c2019-04-15 17:58:45 -0700217# Use this for internal comparisons; that's what canonicalization is
218# for. Don't use it when invoking other components of the build system
219# to avoid confusing and hard to trace inconsistencies in error messages
220# and logs, generated Makefiles, etc. compared to when users invoke these
221# components directly.
222# Note "normalization" is different from canonicalization, see os.path.
223canonical_zephyr_base = os.path.realpath(ZEPHYR_BASE)
224
Andrew Boie3ea78922016-03-24 14:46:00 -0700225sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/"))
226
Anas Nashif83fc06a2019-06-22 11:04:10 -0400227from sanity_chk import scl
228from sanity_chk import expr_parser
229
Andrew Boie6acbe632015-07-17 12:03:52 -0700230VERBOSE = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -0400231
Andrew Boie6acbe632015-07-17 12:03:52 -0700232RELEASE_DATA = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk",
233 "sanity_last_release.csv")
Andrew Boie6acbe632015-07-17 12:03:52 -0700234
235if os.isatty(sys.stdout.fileno()):
236 TERMINAL = True
237 COLOR_NORMAL = '\033[0m'
238 COLOR_RED = '\033[91m'
239 COLOR_GREEN = '\033[92m'
240 COLOR_YELLOW = '\033[93m'
241else:
242 TERMINAL = False
243 COLOR_NORMAL = ""
244 COLOR_RED = ""
245 COLOR_GREEN = ""
246 COLOR_YELLOW = ""
247
Anas Nashif7a361b82019-12-06 11:37:40 -0500248logger = logging.getLogger('sanitycheck')
Anas Nashifd9882382019-12-12 09:58:28 -0500249# coloredlogs.install(level='INFO', logger=logger, fmt="%(levelname)s %(message)s")
Anas Nashif7a361b82019-12-06 11:37:40 -0500250logger.setLevel(logging.DEBUG)
251
252
Anas Nashifd9882382019-12-12 09:58:28 -0500253# log_format = "%(levelname)s %(message)s"
254# logging.basicConfig(format=log_format, level=logging.INFO)
Anas Nashif7a361b82019-12-06 11:37:40 -0500255
Anas Nashif45a97862019-01-09 08:46:42 -0500256class CMakeCacheEntry:
257 '''Represents a CMake cache entry.
258
259 This class understands the type system in a CMakeCache.txt, and
260 converts the following cache types to Python types:
261
262 Cache Type Python type
263 ---------- -------------------------------------------
264 FILEPATH str
265 PATH str
266 STRING str OR list of str (if ';' is in the value)
267 BOOL bool
268 INTERNAL str OR list of str (if ';' is in the value)
269 ---------- -------------------------------------------
270 '''
271
272 # Regular expression for a cache entry.
273 #
274 # CMake variable names can include escape characters, allowing a
275 # wider set of names than is easy to match with a regular
276 # expression. To be permissive here, use a non-greedy match up to
277 # the first colon (':'). This breaks if the variable name has a
278 # colon inside, but it's good enough.
279 CACHE_ENTRY = re.compile(
280 r'''(?P<name>.*?) # name
281 :(?P<type>FILEPATH|PATH|STRING|BOOL|INTERNAL) # type
282 =(?P<value>.*) # value
283 ''', re.X)
284
285 @classmethod
286 def _to_bool(cls, val):
287 # Convert a CMake BOOL string into a Python bool.
288 #
289 # "True if the constant is 1, ON, YES, TRUE, Y, or a
290 # non-zero number. False if the constant is 0, OFF, NO,
291 # FALSE, N, IGNORE, NOTFOUND, the empty string, or ends in
292 # the suffix -NOTFOUND. Named boolean constants are
293 # case-insensitive. If the argument is not one of these
294 # constants, it is treated as a variable."
295 #
296 # https://cmake.org/cmake/help/v3.0/command/if.html
297 val = val.upper()
298 if val in ('ON', 'YES', 'TRUE', 'Y'):
299 return 1
300 elif val in ('OFF', 'NO', 'FALSE', 'N', 'IGNORE', 'NOTFOUND', ''):
301 return 0
302 elif val.endswith('-NOTFOUND'):
303 return 0
304 else:
305 try:
306 v = int(val)
307 return v != 0
308 except ValueError as exc:
309 raise ValueError('invalid bool {}'.format(val)) from exc
310
311 @classmethod
312 def from_line(cls, line, line_no):
313 # Comments can only occur at the beginning of a line.
314 # (The value of an entry could contain a comment character).
315 if line.startswith('//') or line.startswith('#'):
316 return None
317
318 # Whitespace-only lines do not contain cache entries.
319 if not line.strip():
320 return None
321
322 m = cls.CACHE_ENTRY.match(line)
323 if not m:
324 return None
325
326 name, type_, value = (m.group(g) for g in ('name', 'type', 'value'))
327 if type_ == 'BOOL':
328 try:
329 value = cls._to_bool(value)
330 except ValueError as exc:
331 args = exc.args + ('on line {}: {}'.format(line_no, line),)
332 raise ValueError(args) from exc
Anas Nashifd9882382019-12-12 09:58:28 -0500333 elif type_ in ['STRING', 'INTERNAL']:
Anas Nashif45a97862019-01-09 08:46:42 -0500334 # If the value is a CMake list (i.e. is a string which
335 # contains a ';'), convert to a Python list.
336 if ';' in value:
337 value = value.split(';')
338
339 return CMakeCacheEntry(name, value)
340
341 def __init__(self, name, value):
342 self.name = name
343 self.value = value
344
345 def __str__(self):
346 fmt = 'CMakeCacheEntry(name={}, value={})'
347 return fmt.format(self.name, self.value)
348
349
350class CMakeCache:
351 '''Parses and represents a CMake cache file.'''
352
353 @staticmethod
354 def from_file(cache_file):
355 return CMakeCache(cache_file)
356
357 def __init__(self, cache_file):
358 self.cache_file = cache_file
359 self.load(cache_file)
360
361 def load(self, cache_file):
362 entries = []
363 with open(cache_file, 'r') as cache:
364 for line_no, line in enumerate(cache):
365 entry = CMakeCacheEntry.from_line(line, line_no)
366 if entry:
367 entries.append(entry)
368 self._entries = OrderedDict((e.name, e) for e in entries)
369
370 def get(self, name, default=None):
371 entry = self._entries.get(name)
372 if entry is not None:
373 return entry.value
374 else:
375 return default
376
377 def get_list(self, name, default=None):
378 if default is None:
379 default = []
380 entry = self._entries.get(name)
381 if entry is not None:
382 value = entry.value
383 if isinstance(value, list):
384 return value
385 elif isinstance(value, str):
386 return [value] if value else []
387 else:
388 msg = 'invalid value {} type {}'
389 raise RuntimeError(msg.format(value, type(value)))
390 else:
391 return default
392
393 def __contains__(self, name):
394 return name in self._entries
395
396 def __getitem__(self, name):
397 return self._entries[name].value
398
399 def __setitem__(self, name, entry):
400 if not isinstance(entry, CMakeCacheEntry):
401 msg = 'improper type {} for value {}, expecting CMakeCacheEntry'
402 raise TypeError(msg.format(type(entry), entry))
403 self._entries[name] = entry
404
405 def __delitem__(self, name):
406 del self._entries[name]
407
408 def __iter__(self):
409 return iter(self._entries.values())
410
Anas Nashif11ee5252019-12-04 12:59:10 -0500411
Andrew Boie6acbe632015-07-17 12:03:52 -0700412class SanityCheckException(Exception):
413 pass
414
Anas Nashif3ba1d432017-12-05 15:28:44 -0500415
Andrew Boie6acbe632015-07-17 12:03:52 -0700416class SanityRuntimeError(SanityCheckException):
417 pass
418
Anas Nashif11ee5252019-12-04 12:59:10 -0500419
Andrew Boie6acbe632015-07-17 12:03:52 -0700420class ConfigurationError(SanityCheckException):
421 def __init__(self, cfile, message):
Anas Nashif83fc06a2019-06-22 11:04:10 -0400422 SanityCheckException.__init__(self, cfile + ": " + message)
Andrew Boie6acbe632015-07-17 12:03:52 -0700423
Anas Nashif11ee5252019-12-04 12:59:10 -0500424
Anas Nashif83fc06a2019-06-22 11:04:10 -0400425class BuildError(SanityCheckException):
Andrew Boie6acbe632015-07-17 12:03:52 -0700426 pass
427
Anas Nashif3ba1d432017-12-05 15:28:44 -0500428
Anas Nashif83fc06a2019-06-22 11:04:10 -0400429class ExecutionError(SanityCheckException):
Andrew Boie6acbe632015-07-17 12:03:52 -0700430 pass
431
Anas Nashif11ee5252019-12-04 12:59:10 -0500432
Anas Nashif576be982017-12-23 20:20:27 -0500433class HarnessImporter:
434
435 def __init__(self, name):
436 sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/sanity_chk"))
437 module = __import__("harness")
438 if name:
439 my_class = getattr(module, name)
440 else:
441 my_class = getattr(module, "Test")
442
443 self.instance = my_class()
Anas Nashif3ba1d432017-12-05 15:28:44 -0500444
Anas Nashif11ee5252019-12-04 12:59:10 -0500445
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300446class Handler:
Anas Nashifd18ec532019-04-11 23:20:39 -0400447 def __init__(self, instance, type_str="build"):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300448 """Constructor
449
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300450 """
451 self.lock = threading.Lock()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400452
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300453 self.state = "waiting"
Anas Nashif13773752018-07-06 18:20:23 -0500454 self.run = False
Anas Nashif83fc06a2019-06-22 11:04:10 -0400455 self.duration = 0
Anas Nashifd18ec532019-04-11 23:20:39 -0400456 self.type_str = type_str
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300457
Anas Nashifdf7ee612018-07-07 06:09:01 -0500458 self.binary = None
Jan Kowalewski265895b2019-01-07 16:40:24 +0100459 self.pid_fn = None
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500460 self.call_make_run = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500461
Anas Nashifd3384fb2018-02-22 06:44:16 -0600462 self.name = instance.name
463 self.instance = instance
Anas Nashif83fc06a2019-06-22 11:04:10 -0400464 self.timeout = instance.testcase.timeout
465 self.sourcedir = instance.testcase.source_dir
466 self.build_dir = instance.build_dir
467 self.log = os.path.join(self.build_dir, "handler.log")
Anas Nashifd3384fb2018-02-22 06:44:16 -0600468 self.returncode = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -0400469 self.set_state("running", self.duration)
Anas Nashifd3384fb2018-02-22 06:44:16 -0600470
Anas Nashif83fc06a2019-06-22 11:04:10 -0400471 self.args = []
472
473 def set_state(self, state, duration):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300474 self.lock.acquire()
475 self.state = state
Anas Nashif83fc06a2019-06-22 11:04:10 -0400476 self.duration = duration
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300477 self.lock.release()
478
479 def get_state(self):
480 self.lock.acquire()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400481 ret = (self.state, self.duration)
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300482 self.lock.release()
483 return ret
484
Anas Nashif83fc06a2019-06-22 11:04:10 -0400485 def record(self, harness):
486 if harness.recording:
Anas Nashif9e7a2cd2019-12-05 13:07:27 -0500487 filename = os.path.join(self.build_dir, "recording.csv")
Anas Nashif83fc06a2019-06-22 11:04:10 -0400488 with open(filename, "at") as csvfile:
489 cw = csv.writer(csvfile, harness.fieldnames, lineterminator=os.linesep)
490 cw.writerow(harness.fieldnames)
491 for instance in harness.recording:
492 cw.writerow(instance)
493
Anas Nashif11ee5252019-12-04 12:59:10 -0500494
Anas Nashifdf7ee612018-07-07 06:09:01 -0500495class BinaryHandler(Handler):
Anas Nashifd18ec532019-04-11 23:20:39 -0400496 def __init__(self, instance, type_str):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500497 """Constructor
498
499 @param instance Test Instance
500 """
Anas Nashifd18ec532019-04-11 23:20:39 -0400501 super().__init__(instance, type_str)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500502
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100503 self.terminated = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500504
Anas Nashif6c0e1702019-12-05 15:24:52 -0500505 # Tool options
506 self.valgrind = False
507 self.lsan = False
508 self.asan = False
509 self.coverage = False
510
Jan Kowalewski265895b2019-01-07 16:40:24 +0100511 def try_kill_process_by_pid(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -0400512 if self.pid_fn:
Jan Kowalewski265895b2019-01-07 16:40:24 +0100513 pid = int(open(self.pid_fn).read())
514 os.unlink(self.pid_fn)
515 self.pid_fn = None # clear so we don't try to kill the binary twice
516 try:
517 os.kill(pid, signal.SIGTERM)
518 except ProcessLookupError:
519 pass
520
Kumar Gala34b1ef82019-12-12 04:38:42 -0600521 def terminate(self, proc):
522 # encapsulate terminate functionality so we do it consistently where ever
523 # we might want to terminate the proc. We need try_kill_process_by_pid
524 # because of both how newer ninja (1.6.0 or greater) and .NET / renode
525 # work. Newer ninja's don't seem to pass SIGTERM down to the children
526 # so we need to use try_kill_process_by_pid.
527 self.try_kill_process_by_pid()
528 proc.terminate()
529 self.terminated = True
530
Anas Nashifdf7ee612018-07-07 06:09:01 -0500531 def _output_reader(self, proc, harness):
532 log_out_fp = open(self.log, "wt")
533 for line in iter(proc.stdout.readline, b''):
Anas Nashif7a361b82019-12-06 11:37:40 -0500534 logger.debug("OUTPUT: {0}".format(line.decode('utf-8').rstrip()))
Anas Nashifdf7ee612018-07-07 06:09:01 -0500535 log_out_fp.write(line.decode('utf-8'))
536 log_out_fp.flush()
537 harness.handle(line.decode('utf-8').rstrip())
538 if harness.state:
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100539 try:
Anas Nashifd9882382019-12-12 09:58:28 -0500540 # POSIX arch based ztests end on their own,
541 # so let's give it up to 100ms to do so
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100542 proc.wait(0.1)
543 except subprocess.TimeoutExpired:
Kumar Gala34b1ef82019-12-12 04:38:42 -0600544 self.terminate(proc)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500545 break
546
547 log_out_fp.close()
548
549 def handle(self):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500550
Anas Nashif83fc06a2019-06-22 11:04:10 -0400551 harness_name = self.instance.testcase.harness.capitalize()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500552 harness_import = HarnessImporter(harness_name)
553 harness = harness_import.instance
554 harness.configure(self.instance)
555
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500556 if self.call_make_run:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400557 command = [get_generator()[0], "run"]
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500558 else:
559 command = [self.binary]
Anas Nashifdf7ee612018-07-07 06:09:01 -0500560
Anas Nashifc1ea4522019-10-11 07:32:45 -0700561 run_valgrind = False
Anas Nashif6c0e1702019-12-05 15:24:52 -0500562 if self.valgrind and shutil.which("valgrind"):
Anas Nashifdf7ee612018-07-07 06:09:01 -0500563 command = ["valgrind", "--error-exitcode=2",
Alberto Escolar Piedras3980e0c2018-12-12 17:41:04 +0100564 "--leak-check=full",
Anas Nashifd9882382019-12-12 09:58:28 -0500565 "--suppressions=" + ZEPHYR_BASE + "/scripts/valgrind.supp",
566 "--log-file=" + self.build_dir + "/valgrind.log"
Alberto Escolar Piedras3980e0c2018-12-12 17:41:04 +0100567 ] + command
Anas Nashifc1ea4522019-10-11 07:32:45 -0700568 run_valgrind = True
Anas Nashifdf7ee612018-07-07 06:09:01 -0500569
Anas Nashif7a361b82019-12-06 11:37:40 -0500570 logger.debug("Spawning process: " +
Anas Nashifd9882382019-12-12 09:58:28 -0500571 " ".join(shlex.quote(word) for word in command) + os.linesep +
572 "in directory: " + self.build_dir)
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200573
Anas Nashif83fc06a2019-06-22 11:04:10 -0400574 start_time = time.time()
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200575
Anas Nashif89c83042019-11-05 05:55:39 -0800576 env = os.environ.copy()
Anas Nashif6c0e1702019-12-05 15:24:52 -0500577 if self.asan:
Jan Van Winkel21212f32019-09-12 00:03:35 +0200578 env["ASAN_OPTIONS"] = "log_path=stdout:" + \
Anas Nashifd9882382019-12-12 09:58:28 -0500579 env.get("ASAN_OPTIONS", "")
Anas Nashif6c0e1702019-12-05 15:24:52 -0500580 if not self.lsan:
Jan Van Winkel21212f32019-09-12 00:03:35 +0200581 env["ASAN_OPTIONS"] += "detect_leaks=0"
582 with subprocess.Popen(command, stdout=subprocess.PIPE,
Anas Nashifd9882382019-12-12 09:58:28 -0500583 stderr=subprocess.PIPE, cwd=self.build_dir, env=env) as proc:
Anas Nashif7a361b82019-12-06 11:37:40 -0500584 logger.debug("Spawning BinaryHandler Thread for %s" % self.name)
Anas Nashifd9882382019-12-12 09:58:28 -0500585 t = threading.Thread(target=self._output_reader, args=(proc, harness,), daemon=True)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500586 t.start()
587 t.join(self.timeout)
588 if t.is_alive():
Kumar Gala34b1ef82019-12-12 04:38:42 -0600589 self.terminate(proc)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500590 t.join()
Anas Nashifdf7ee612018-07-07 06:09:01 -0500591 proc.wait()
592 self.returncode = proc.returncode
Anas Nashifdf7ee612018-07-07 06:09:01 -0500593
Anas Nashif83fc06a2019-06-22 11:04:10 -0400594 handler_time = time.time() - start_time
Alberto Escolar Piedras649368c2019-07-02 09:59:35 +0200595
Anas Nashif6c0e1702019-12-05 15:24:52 -0500596 if self.coverage:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400597 subprocess.call(["GCOV_PREFIX=" + self.build_dir,
Anas Nashifd9882382019-12-12 09:58:28 -0500598 "gcov", self.sourcedir, "-b", "-s", self.build_dir], shell=True)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500599
Jan Kowalewski265895b2019-01-07 16:40:24 +0100600 self.try_kill_process_by_pid()
601
Anas Nashif83fc06a2019-06-22 11:04:10 -0400602 # FIXME: This is needed when killing the simulator, the console is
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500603 # garbled and needs to be reset. Did not find a better way to do that.
604
605 subprocess.call(["stty", "sane"])
Anas Nashifdf7ee612018-07-07 06:09:01 -0500606 self.instance.results = harness.tests
Anas Nashifc1ea4522019-10-11 07:32:45 -0700607
Anas Nashif83fc06a2019-06-22 11:04:10 -0400608 if not self.terminated and self.returncode != 0:
Anas Nashifd9882382019-12-12 09:58:28 -0500609 # When a process is killed, the default handler returns 128 + SIGTERM
610 # so in that case the return code itself is not meaningful
Anas Nashiff72d1902019-10-18 17:20:44 -0400611 self.set_state("failed", handler_time)
Anas Nashifc1ea4522019-10-11 07:32:45 -0700612 self.instance.reason = "Handler Error"
613 elif run_valgrind and self.returncode == 2:
614 self.set_state("failed", handler_time)
615 self.instance.reason = "Valgrind error"
Alberto Escolar Piedras661bc822018-12-11 15:13:05 +0100616 elif harness.state:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400617 self.set_state(harness.state, handler_time)
Anas Nashifdf7ee612018-07-07 06:09:01 -0500618 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400619 self.set_state("timeout", handler_time)
620 self.instance.reason = "Handler timeout"
621
Anas Nashif83fc06a2019-06-22 11:04:10 -0400622 self.record(harness)
Anas Nashif73440ea2018-02-19 10:57:03 -0600623
Anas Nashif11ee5252019-12-04 12:59:10 -0500624
Anas Nashif73440ea2018-02-19 10:57:03 -0600625class DeviceHandler(Handler):
626
Anas Nashifd18ec532019-04-11 23:20:39 -0400627 def __init__(self, instance, type_str):
Anas Nashif73440ea2018-02-19 10:57:03 -0600628 """Constructor
629
630 @param instance Test Instance
631 """
Anas Nashifd18ec532019-04-11 23:20:39 -0400632 super().__init__(instance, type_str)
Anas Nashif73440ea2018-02-19 10:57:03 -0600633
Anas Nashif83fc06a2019-06-22 11:04:10 -0400634 self.suite = None
635
Marti Bolivar5591ca22019-02-07 15:53:39 -0700636 def monitor_serial(self, ser, halt_fileno, harness):
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500637 log_out_fp = open(self.log, "wt")
Anas Nashif73440ea2018-02-19 10:57:03 -0600638
Marti Bolivar5591ca22019-02-07 15:53:39 -0700639 ser_fileno = ser.fileno()
640 readlist = [halt_fileno, ser_fileno]
641
Anas Nashif73440ea2018-02-19 10:57:03 -0600642 while ser.isOpen():
Marti Bolivar5591ca22019-02-07 15:53:39 -0700643 readable, _, _ = select.select(readlist, [], [], self.timeout)
644
645 if halt_fileno in readable:
Anas Nashif7a361b82019-12-06 11:37:40 -0500646 logger.debug('halted')
Marti Bolivar5591ca22019-02-07 15:53:39 -0700647 ser.close()
648 break
649 if ser_fileno not in readable:
Anas Nashifd9882382019-12-12 09:58:28 -0500650 continue # Timeout.
Marti Bolivar5591ca22019-02-07 15:53:39 -0700651
652 serial_line = None
Anas Nashif61e21632018-04-08 13:30:16 -0500653 try:
654 serial_line = ser.readline()
655 except TypeError:
656 pass
Anas Nashif83fc06a2019-06-22 11:04:10 -0400657 except serial.SerialException:
Anas Nashif59237602019-03-03 10:36:35 -0500658 ser.close()
659 break
Anas Nashif61e21632018-04-08 13:30:16 -0500660
Marti Bolivar5591ca22019-02-07 15:53:39 -0700661 # Just because ser_fileno has data doesn't mean an entire line
662 # is available yet.
Anas Nashif73440ea2018-02-19 10:57:03 -0600663 if serial_line:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600664 sl = serial_line.decode('utf-8', 'ignore')
Anas Nashif7a361b82019-12-06 11:37:40 -0500665 logger.debug("DEVICE: {0}".format(sl.rstrip()))
Anas Nashifd3384fb2018-02-22 06:44:16 -0600666
667 log_out_fp.write(sl)
668 log_out_fp.flush()
669 harness.handle(sl.rstrip())
Marti Bolivar5591ca22019-02-07 15:53:39 -0700670
Anas Nashif73440ea2018-02-19 10:57:03 -0600671 if harness.state:
672 ser.close()
673 break
674
675 log_out_fp.close()
676
Anas Nashif83fc06a2019-06-22 11:04:10 -0400677 def device_is_available(self, device):
678 for i in self.suite.connected_hardware:
Anas Nashif5f908822019-11-25 08:19:25 -0500679 if i['platform'] == device and i['available'] and i['serial']:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400680 return True
681
682 return False
683
684 def get_available_device(self, device):
685 for i in self.suite.connected_hardware:
Anas Nashif5f908822019-11-25 08:19:25 -0500686 if i['platform'] == device and i['available'] and i['serial']:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400687 i['available'] = False
688 i['counter'] += 1
689 return i
690
691 return None
692
693 def make_device_available(self, serial):
694 with hw_map_local:
695 for i in self.suite.connected_hardware:
696 if i['serial'] == serial:
697 i['available'] = True
698
Anas Nashif73440ea2018-02-19 10:57:03 -0600699 def handle(self):
700 out_state = "failed"
701
Anas Nashif83fc06a2019-06-22 11:04:10 -0400702 if options.west_flash:
703 command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir]
704 if options.west_runner:
Michael Scott421ce462019-06-18 09:37:46 -0700705 command.append("--runner")
706 command.append(options.west_runner)
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200707 # There are three ways this option is used.
Andy Doan79c48842019-02-08 10:09:04 -0600708 # 1) bare: --west-flash
709 # This results in options.west_flash == []
710 # 2) with a value: --west-flash="--board-id=42"
711 # This results in options.west_flash == "--board-id=42"
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200712 # 3) Multiple values: --west-flash="--board-id=42,--erase"
713 # This results in options.west_flash == "--board-id=42 --erase"
Andy Doan79c48842019-02-08 10:09:04 -0600714 if options.west_flash != []:
715 command.append('--')
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +0200716 command.extend(options.west_flash.split(','))
Anas Nashifd3384fb2018-02-22 06:44:16 -0600717 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400718 command = [get_generator()[0], "-C", self.build_dir, "flash"]
Anas Nashifd3384fb2018-02-22 06:44:16 -0600719
Anas Nashif83fc06a2019-06-22 11:04:10 -0400720 while not self.device_is_available(self.instance.platform.name):
Anas Nashifebf8dae2019-12-16 09:22:21 -0500721 #logger.debug("Waiting for device to become available...")
Anas Nashif83fc06a2019-06-22 11:04:10 -0400722 time.sleep(1)
723
724 hardware = self.get_available_device(self.instance.platform.name)
725
726 runner = hardware.get('runner', None)
727 if runner:
Peter Bigotda738482019-11-21 11:55:26 -0600728 board_id = hardware.get("probe_id", hardware.get("id", None))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400729 product = hardware.get("product", None)
730 command = ["west", "flash", "--skip-rebuild", "-d", self.build_dir]
731 command.append("--runner")
732 command.append(hardware.get('runner', None))
733 if runner == "pyocd":
734 command.append("--board-id")
735 command.append(board_id)
736 elif runner == "nrfjprog":
737 command.append('--')
738 command.append("--snr")
739 command.append(board_id)
740 elif runner == "openocd" and product == "STM32 STLink":
741 command.append('--')
742 command.append("--cmd-pre-init")
Anas Nashifd9882382019-12-12 09:58:28 -0500743 command.append("hla_serial %s" % (board_id))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400744 elif runner == "openocd" and product == "EDBG CMSIS-DAP":
745 command.append('--')
746 command.append("--cmd-pre-init")
Anas Nashifd9882382019-12-12 09:58:28 -0500747 command.append("cmsis_dap_serial %s" % (board_id))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400748 elif runner == "jlink":
Anas Nashifd9882382019-12-12 09:58:28 -0500749 command.append("--tool-opt=-SelectEmuBySN %s" % (board_id))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400750
751 serial_device = hardware['serial']
752
753 try:
754 ser = serial.Serial(
Anas Nashifd9882382019-12-12 09:58:28 -0500755 serial_device,
756 baudrate=115200,
757 parity=serial.PARITY_NONE,
758 stopbits=serial.STOPBITS_ONE,
759 bytesize=serial.EIGHTBITS,
760 timeout=self.timeout
761 )
Anas Nashif83fc06a2019-06-22 11:04:10 -0400762 except serial.SerialException as e:
763 self.set_state("failed", 0)
Anas Nashifd9882382019-12-12 09:58:28 -0500764 logger.error("Serial device error: %s" % (str(e)))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400765 self.make_device_available(serial_device)
766 return
Anas Nashif73440ea2018-02-19 10:57:03 -0600767
768 ser.flush()
769
Anas Nashif83fc06a2019-06-22 11:04:10 -0400770 harness_name = self.instance.testcase.harness.capitalize()
Anas Nashif73440ea2018-02-19 10:57:03 -0600771 harness_import = HarnessImporter(harness_name)
772 harness = harness_import.instance
773 harness.configure(self.instance)
Anas Nashif83fc06a2019-06-22 11:04:10 -0400774 read_pipe, write_pipe = os.pipe()
775 start_time = time.time()
Anas Nashif73440ea2018-02-19 10:57:03 -0600776
Marti Bolivar5591ca22019-02-07 15:53:39 -0700777 t = threading.Thread(target=self.monitor_serial, daemon=True,
Anas Nashifd9882382019-12-12 09:58:28 -0500778 args=(ser, read_pipe, harness))
Anas Nashif73440ea2018-02-19 10:57:03 -0600779 t.start()
780
Anas Nashif33ed7aa2019-11-25 12:18:23 -0500781 d_log = "{}/device.log".format(self.instance.build_dir)
Anas Nashif7a361b82019-12-06 11:37:40 -0500782 logger.debug('Flash command: %s', command)
Anas Nashif61e21632018-04-08 13:30:16 -0500783 try:
Anas Nashif33ed7aa2019-11-25 12:18:23 -0500784 stdout = stderr = None
785 with subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE) as proc:
786 try:
787 (stdout, stderr) = proc.communicate(timeout=30)
Anas Nashiffa8085e2019-12-09 16:42:58 -0500788 logger.debug(stdout.decode())
789
Anas Nashif33ed7aa2019-11-25 12:18:23 -0500790 if proc.returncode != 0:
791 self.instance.reason = "Device issue (Flash?)"
792 with open(d_log, "w") as dlog_fp:
793 dlog_fp.write(stderr.decode())
794 except subprocess.TimeoutExpired:
795 proc.kill()
Anas Nashifd9882382019-12-12 09:58:28 -0500796 (stdout, stderr) = proc.communicate()
Anas Nashif33ed7aa2019-11-25 12:18:23 -0500797 self.instance.reason = "Device issue (Timeout)"
798
799 with open(d_log, "w") as dlog_fp:
800 dlog_fp.write(stderr.decode())
Anas Nashif83fc06a2019-06-22 11:04:10 -0400801
Anas Nashif61e21632018-04-08 13:30:16 -0500802 except subprocess.CalledProcessError:
Anas Nashifd9882382019-12-12 09:58:28 -0500803 os.write(write_pipe, b'x') # halt the thread
Anas Nashif73440ea2018-02-19 10:57:03 -0600804
805 t.join(self.timeout)
806 if t.is_alive():
807 out_state = "timeout"
Anas Nashif73440ea2018-02-19 10:57:03 -0600808
809 if ser.isOpen():
810 ser.close()
811
Anas Nashifd3384fb2018-02-22 06:44:16 -0600812 if out_state == "timeout":
Anas Nashif83fc06a2019-06-22 11:04:10 -0400813 for c in self.instance.testcase.cases:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600814 if c not in harness.tests:
815 harness.tests[c] = "BLOCK"
Anas Nashif61e21632018-04-08 13:30:16 -0500816
Anas Nashif83fc06a2019-06-22 11:04:10 -0400817 handler_time = time.time() - start_time
818
Anas Nashif61e21632018-04-08 13:30:16 -0500819 self.instance.results = harness.tests
Anas Nashif73440ea2018-02-19 10:57:03 -0600820 if harness.state:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400821 self.set_state(harness.state, handler_time)
Anas Nashif73440ea2018-02-19 10:57:03 -0600822 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400823 self.set_state(out_state, handler_time)
Anas Nashif73440ea2018-02-19 10:57:03 -0600824
Anas Nashif83fc06a2019-06-22 11:04:10 -0400825 self.make_device_available(serial_device)
Anas Nashif3ba1d432017-12-05 15:28:44 -0500826
Anas Nashif3a00d0c2019-11-12 11:07:15 -0500827 self.record(harness)
828
Anas Nashif11ee5252019-12-04 12:59:10 -0500829
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300830class QEMUHandler(Handler):
Andrew Boie6acbe632015-07-17 12:03:52 -0700831 """Spawns a thread to monitor QEMU output from pipes
832
Anas Nashif3ac7b3a2017-08-03 10:03:02 -0400833 We pass QEMU_PIPE to 'make run' and monitor the pipes for output.
Andrew Boie6acbe632015-07-17 12:03:52 -0700834 We need to do this as once qemu starts, it runs forever until killed.
835 Test cases emit special messages to the console as they run, we check
836 for these to collect whether the test passed or failed.
837 """
Andrew Boie6acbe632015-07-17 12:03:52 -0700838
Anas Nashif83fc06a2019-06-22 11:04:10 -0400839 def __init__(self, instance, type_str):
840 """Constructor
841
842 @param instance Test instance
843 """
844
845 super().__init__(instance, type_str)
846 self.fifo_fn = os.path.join(instance.build_dir, "qemu-fifo")
847
848 self.pid_fn = os.path.join(instance.build_dir, "qemu.pid")
849
Andrew Boie6acbe632015-07-17 12:03:52 -0700850 @staticmethod
Anas Nashif576be982017-12-23 20:20:27 -0500851 def _thread(handler, timeout, outdir, logfile, fifo_fn, pid_fn, results, harness):
Andrew Boie6acbe632015-07-17 12:03:52 -0700852 fifo_in = fifo_fn + ".in"
853 fifo_out = fifo_fn + ".out"
854
855 # These in/out nodes are named from QEMU's perspective, not ours
856 if os.path.exists(fifo_in):
857 os.unlink(fifo_in)
858 os.mkfifo(fifo_in)
859 if os.path.exists(fifo_out):
860 os.unlink(fifo_out)
861 os.mkfifo(fifo_out)
862
863 # We don't do anything with out_fp but we need to open it for
864 # writing so that QEMU doesn't block, due to the way pipes work
865 out_fp = open(fifo_in, "wb")
866 # Disable internal buffering, we don't
867 # want read() or poll() to ever block if there is data in there
868 in_fp = open(fifo_out, "rb", buffering=0)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800869 log_out_fp = open(logfile, "wt")
Andrew Boie6acbe632015-07-17 12:03:52 -0700870
871 start_time = time.time()
872 timeout_time = start_time + timeout
873 p = select.poll()
874 p.register(in_fp, select.POLLIN)
Anas Nashif39ae72b2018-08-29 01:45:38 -0400875 out_state = None
Andrew Boie6acbe632015-07-17 12:03:52 -0700876
Andrew Boie6acbe632015-07-17 12:03:52 -0700877 line = ""
Anas Nashif74dbe332019-01-17 17:11:37 -0500878 timeout_extended = False
Andrew Boie6acbe632015-07-17 12:03:52 -0700879 while True:
880 this_timeout = int((timeout_time - time.time()) * 1000)
881 if this_timeout < 0 or not p.poll(this_timeout):
Anas Nashif39ae72b2018-08-29 01:45:38 -0400882 if not out_state:
883 out_state = "timeout"
Andrew Boie6acbe632015-07-17 12:03:52 -0700884 break
885
Genaro Saucedo Tejada2968b362016-08-09 15:11:53 -0500886 try:
887 c = in_fp.read(1).decode("utf-8")
888 except UnicodeDecodeError:
889 # Test is writing something weird, fail
890 out_state = "unexpected byte"
891 break
892
Andrew Boie6acbe632015-07-17 12:03:52 -0700893 if c == "":
894 # EOF, this shouldn't happen unless QEMU crashes
895 out_state = "unexpected eof"
896 break
897 line = line + c
898 if c != "\n":
899 continue
900
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300901 # line contains a full line of data output from QEMU
Andrew Boie6acbe632015-07-17 12:03:52 -0700902 log_out_fp.write(line)
903 log_out_fp.flush()
904 line = line.strip()
Anas Nashif7a361b82019-12-06 11:37:40 -0500905 logger.debug("QEMU: %s" % line)
Andrew Boie6acbe632015-07-17 12:03:52 -0700906
Anas Nashif576be982017-12-23 20:20:27 -0500907 harness.handle(line)
908 if harness.state:
Anas Nashif39ae72b2018-08-29 01:45:38 -0400909 # if we have registered a fail make sure the state is not
910 # overridden by a false success message coming from the
911 # testsuite
912 if out_state != 'failed':
913 out_state = harness.state
914
915 # if we get some state, that means test is doing well, we reset
Marc Herbert9e573382019-07-03 12:49:42 -0700916 # the timeout and wait for 2 more seconds to catch anything
917 # printed late. We wait much longer if code
Andrew Boie095b82a2019-07-02 17:03:48 -0700918 # coverage is enabled since dumping this information can
919 # take some time.
Anas Nashiff29087e2019-01-25 09:37:38 -0500920 if not timeout_extended or harness.capture_coverage:
Anas Nashifd9882382019-12-12 09:58:28 -0500921 timeout_extended = True
Anas Nashiff29087e2019-01-25 09:37:38 -0500922 if harness.capture_coverage:
Andrew Boied62e2292019-07-01 15:01:48 -0700923 timeout_time = time.time() + 30
Anas Nashiff29087e2019-01-25 09:37:38 -0500924 else:
Anas Nashif74dbe332019-01-17 17:11:37 -0500925 timeout_time = time.time() + 2
Andrew Boie6acbe632015-07-17 12:03:52 -0700926 line = ""
927
Anas Nashif83fc06a2019-06-22 11:04:10 -0400928 handler.record(harness)
929
930 handler_time = time.time() - start_time
Anas Nashif7a361b82019-12-06 11:37:40 -0500931 logger.debug("QEMU complete (%s) after %f seconds" %
Anas Nashifd9882382019-12-12 09:58:28 -0500932 (out_state, handler_time))
Anas Nashif83fc06a2019-06-22 11:04:10 -0400933 handler.set_state(out_state, handler_time)
Andrew Boie6acbe632015-07-17 12:03:52 -0700934
935 log_out_fp.close()
936 out_fp.close()
937 in_fp.close()
Anas Nashifd6476ee2019-04-11 11:40:09 -0400938 if os.path.exists(pid_fn):
939 pid = int(open(pid_fn).read())
940 os.unlink(pid_fn)
Andrew Boie6acbe632015-07-17 12:03:52 -0700941
Anas Nashifd6476ee2019-04-11 11:40:09 -0400942 try:
943 if pid:
944 os.kill(pid, signal.SIGTERM)
945 except ProcessLookupError:
946 # Oh well, as long as it's dead! User probably sent Ctrl-C
947 pass
Andrew Boie08ce5a52016-02-22 13:28:10 -0800948
Andrew Boie6acbe632015-07-17 12:03:52 -0700949 os.unlink(fifo_in)
950 os.unlink(fifo_out)
951
Anas Nashif83fc06a2019-06-22 11:04:10 -0400952 def handle(self):
Andrew Boie6acbe632015-07-17 12:03:52 -0700953 self.results = {}
Anas Nashif13773752018-07-06 18:20:23 -0500954 self.run = True
Andrew Boie6acbe632015-07-17 12:03:52 -0700955
956 # We pass this to QEMU which looks for fifos with .in and .out
957 # suffixes.
Anas Nashif83fc06a2019-06-22 11:04:10 -0400958 self.fifo_fn = os.path.join(self.instance.build_dir, "qemu-fifo")
Andrew Boie6acbe632015-07-17 12:03:52 -0700959
Anas Nashif83fc06a2019-06-22 11:04:10 -0400960 self.pid_fn = os.path.join(self.instance.build_dir, "qemu.pid")
Andrew Boie6acbe632015-07-17 12:03:52 -0700961 if os.path.exists(self.pid_fn):
962 os.unlink(self.pid_fn)
963
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500964 self.log_fn = self.log
Anas Nashif576be982017-12-23 20:20:27 -0500965
Anas Nashif83fc06a2019-06-22 11:04:10 -0400966 harness_import = HarnessImporter(self.instance.testcase.harness.capitalize())
Anas Nashif576be982017-12-23 20:20:27 -0500967 harness = harness_import.instance
Anas Nashifd3384fb2018-02-22 06:44:16 -0600968 harness.configure(self.instance)
969 self.thread = threading.Thread(name=self.name, target=QEMUHandler._thread,
Anas Nashif83fc06a2019-06-22 11:04:10 -0400970 args=(self, self.timeout, self.build_dir,
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300971 self.log_fn, self.fifo_fn,
Anas Nashif576be982017-12-23 20:20:27 -0500972 self.pid_fn, self.results, harness))
Anas Nashife0a6a0b2018-02-15 20:07:24 -0600973
974 self.instance.results = harness.tests
Andrew Boie6acbe632015-07-17 12:03:52 -0700975 self.thread.daemon = True
Anas Nashif7a361b82019-12-06 11:37:40 -0500976 logger.debug("Spawning QEMUHandler Thread for %s" % self.name)
Andrew Boie6acbe632015-07-17 12:03:52 -0700977 self.thread.start()
Anas Nashif83fc06a2019-06-22 11:04:10 -0400978 subprocess.call(["stty", "sane"])
Andrew Boie6acbe632015-07-17 12:03:52 -0700979
Anas Nashifd9882382019-12-12 09:58:28 -0500980 logger.debug("Running %s (%s)" % (self.name, self.type_str))
Stephanos Ioannidise1827c02019-11-12 14:11:31 +0900981 command = [get_generator()[0]]
982 command += ["-C", self.build_dir, "run"]
983
984 with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.build_dir) as proc:
Anas Nashif7a361b82019-12-06 11:37:40 -0500985 logger.debug("Spawning QEMUHandler Thread for %s" % self.name)
Stephanos Ioannidise1827c02019-11-12 14:11:31 +0900986 proc.wait()
987 self.returncode = proc.returncode
988
989 if self.returncode != 0:
990 self.set_state("failed", 0)
991 self.instance.reason = "Exited with {}".format(self.returncode)
992
Andrew Boie6acbe632015-07-17 12:03:52 -0700993 def get_fifo(self):
994 return self.fifo_fn
995
Anas Nashif11ee5252019-12-04 12:59:10 -0500996
Andrew Boie6acbe632015-07-17 12:03:52 -0700997class SizeCalculator:
Anas Nashif83fc06a2019-06-22 11:04:10 -0400998 alloc_sections = [
999 "bss",
1000 "noinit",
1001 "app_bss",
1002 "app_noinit",
1003 "ccm_bss",
1004 "ccm_noinit"
Anas Nashifd9882382019-12-12 09:58:28 -05001005 ]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001006
1007 rw_sections = [
1008 "datas",
1009 "initlevel",
1010 "exceptions",
1011 "initshell",
1012 "_static_thread_area",
1013 "_k_timer_area",
1014 "_k_mem_slab_area",
1015 "_k_mem_pool_area",
1016 "sw_isr_table",
1017 "_k_sem_area",
1018 "_k_mutex_area",
1019 "app_shmem_regions",
1020 "_k_fifo_area",
1021 "_k_lifo_area",
1022 "_k_stack_area",
1023 "_k_msgq_area",
1024 "_k_mbox_area",
1025 "_k_pipe_area",
1026 "net_if",
1027 "net_if_dev",
1028 "net_stack",
1029 "net_l2_data",
1030 "_k_queue_area",
1031 "_net_buf_pool_area",
1032 "app_datas",
1033 "kobject_data",
1034 "mmu_tables",
1035 "app_pad",
1036 "priv_stacks",
1037 "ccm_data",
1038 "usb_descriptor",
1039 "usb_data", "usb_bos_desc",
1040 'log_backends_sections',
1041 'log_dynamic_sections',
1042 'log_const_sections',
1043 "app_smem",
1044 'shell_root_cmds_sections',
1045 'log_const_sections',
1046 "font_entry_sections",
1047 "priv_stacks_noinit",
1048 "_TEXT_SECTION_NAME_2",
1049 "_GCOV_BSS_SECTION_NAME",
1050 "gcov",
1051 "nocache"
Anas Nashifd9882382019-12-12 09:58:28 -05001052 ]
Anas Nashifdb9592a2018-10-08 10:19:41 -04001053
Andrew Boie73b4ee62015-10-07 11:33:22 -07001054 # These get copied into RAM only on non-XIP
Anas Nashif83fc06a2019-06-22 11:04:10 -04001055 ro_sections = [
1056 "text",
1057 "ctors",
1058 "init_array",
1059 "reset",
1060 "object_access",
1061 "rodata",
1062 "devconfig",
1063 "net_l2",
1064 "vector",
1065 "sw_isr_table",
1066 "_settings_handlers_area",
1067 "_bt_channels_area",
1068 "_bt_br_channels_area",
1069 "_bt_services_area",
1070 "vectors",
1071 "net_socket_register",
1072 "net_ppp_proto"
Anas Nashifd9882382019-12-12 09:58:28 -05001073 ]
Andrew Boie73b4ee62015-10-07 11:33:22 -07001074
Andrew Boie52fef672016-11-29 12:21:59 -08001075 def __init__(self, filename, extra_sections):
Andrew Boie6acbe632015-07-17 12:03:52 -07001076 """Constructor
1077
Andrew Boiebbd670c2015-08-17 13:16:11 -07001078 @param filename Path to the output binary
1079 The <filename> is parsed by objdump to determine section sizes
Andrew Boie6acbe632015-07-17 12:03:52 -07001080 """
Andrew Boie6acbe632015-07-17 12:03:52 -07001081 # Make sure this is an ELF binary
Andrew Boiebbd670c2015-08-17 13:16:11 -07001082 with open(filename, "rb") as f:
Andrew Boie6acbe632015-07-17 12:03:52 -07001083 magic = f.read(4)
1084
Anas Nashifb4bdd662018-08-15 17:12:28 -05001085 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001086 if magic != b'\x7fELF':
Anas Nashifb4bdd662018-08-15 17:12:28 -05001087 raise SanityRuntimeError("%s is not an ELF binary" % filename)
1088 except Exception as e:
1089 print(str(e))
1090 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -07001091
1092 # Search for CONFIG_XIP in the ELF's list of symbols using NM and AWK.
Anas Nashif3ba1d432017-12-05 15:28:44 -05001093 # GREP can not be used as it returns an error if the symbol is not
1094 # found.
1095 is_xip_command = "nm " + filename + \
Anas Nashifd9882382019-12-12 09:58:28 -05001096 " | awk '/CONFIG_XIP/ { print $3 }'"
Anas Nashif3ba1d432017-12-05 15:28:44 -05001097 is_xip_output = subprocess.check_output(
1098 is_xip_command, shell=True, stderr=subprocess.STDOUT).decode(
1099 "utf-8").strip()
Anas Nashifb4bdd662018-08-15 17:12:28 -05001100 try:
1101 if is_xip_output.endswith("no symbols"):
1102 raise SanityRuntimeError("%s has no symbol information" % filename)
1103 except Exception as e:
1104 print(str(e))
1105 sys.exit(2)
1106
Andrew Boie6acbe632015-07-17 12:03:52 -07001107 self.is_xip = (len(is_xip_output) != 0)
1108
Andrew Boiebbd670c2015-08-17 13:16:11 -07001109 self.filename = filename
Andrew Boie73b4ee62015-10-07 11:33:22 -07001110 self.sections = []
1111 self.rom_size = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07001112 self.ram_size = 0
Andrew Boie52fef672016-11-29 12:21:59 -08001113 self.extra_sections = extra_sections
Andrew Boie6acbe632015-07-17 12:03:52 -07001114
1115 self._calculate_sizes()
1116
1117 def get_ram_size(self):
1118 """Get the amount of RAM the application will use up on the device
1119
1120 @return amount of RAM, in bytes
1121 """
Andrew Boie73b4ee62015-10-07 11:33:22 -07001122 return self.ram_size
Andrew Boie6acbe632015-07-17 12:03:52 -07001123
1124 def get_rom_size(self):
1125 """Get the size of the data that this application uses on device's flash
1126
1127 @return amount of ROM, in bytes
1128 """
Andrew Boie73b4ee62015-10-07 11:33:22 -07001129 return self.rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -07001130
1131 def unrecognized_sections(self):
1132 """Get a list of sections inside the binary that weren't recognized
1133
David B. Kinder29963c32017-06-16 12:32:42 -07001134 @return list of unrecognized section names
Andrew Boie6acbe632015-07-17 12:03:52 -07001135 """
1136 slist = []
Andrew Boie73b4ee62015-10-07 11:33:22 -07001137 for v in self.sections:
Andrew Boie6acbe632015-07-17 12:03:52 -07001138 if not v["recognized"]:
Andrew Boie73b4ee62015-10-07 11:33:22 -07001139 slist.append(v["name"])
Andrew Boie6acbe632015-07-17 12:03:52 -07001140 return slist
1141
1142 def _calculate_sizes(self):
1143 """ Calculate RAM and ROM usage by section """
Andrew Boiebbd670c2015-08-17 13:16:11 -07001144 objdump_command = "objdump -h " + self.filename
Anas Nashif3ba1d432017-12-05 15:28:44 -05001145 objdump_output = subprocess.check_output(
1146 objdump_command, shell=True).decode("utf-8").splitlines()
Andrew Boie6acbe632015-07-17 12:03:52 -07001147
1148 for line in objdump_output:
1149 words = line.split()
1150
Anas Nashifd9882382019-12-12 09:58:28 -05001151 if not words: # Skip lines that are too short
Andrew Boie6acbe632015-07-17 12:03:52 -07001152 continue
1153
1154 index = words[0]
Anas Nashifd9882382019-12-12 09:58:28 -05001155 if not index[0].isdigit(): # Skip lines that do not start
1156 continue # with a digit
Andrew Boie6acbe632015-07-17 12:03:52 -07001157
Anas Nashifd9882382019-12-12 09:58:28 -05001158 name = words[1] # Skip lines with section names
1159 if name[0] == '.': # starting with '.'
Andrew Boie6acbe632015-07-17 12:03:52 -07001160 continue
1161
Andrew Boie73b4ee62015-10-07 11:33:22 -07001162 # TODO this doesn't actually reflect the size in flash or RAM as
1163 # it doesn't include linker-imposed padding between sections.
1164 # It is close though.
Andrew Boie6acbe632015-07-17 12:03:52 -07001165 size = int(words[2], 16)
Andrew Boie9882dcd2015-10-07 14:25:51 -07001166 if size == 0:
1167 continue
1168
Andrew Boie73b4ee62015-10-07 11:33:22 -07001169 load_addr = int(words[4], 16)
1170 virt_addr = int(words[3], 16)
Andrew Boie6acbe632015-07-17 12:03:52 -07001171
1172 # Add section to memory use totals (for both non-XIP and XIP scenarios)
Andrew Boie6acbe632015-07-17 12:03:52 -07001173 # Unrecognized section names are not included in the calculations.
Andrew Boie6acbe632015-07-17 12:03:52 -07001174 recognized = True
Andrew Boie73b4ee62015-10-07 11:33:22 -07001175 if name in SizeCalculator.alloc_sections:
1176 self.ram_size += size
1177 stype = "alloc"
1178 elif name in SizeCalculator.rw_sections:
1179 self.ram_size += size
1180 self.rom_size += size
1181 stype = "rw"
1182 elif name in SizeCalculator.ro_sections:
1183 self.rom_size += size
1184 if not self.is_xip:
1185 self.ram_size += size
1186 stype = "ro"
Andrew Boie6acbe632015-07-17 12:03:52 -07001187 else:
Andrew Boie73b4ee62015-10-07 11:33:22 -07001188 stype = "unknown"
Andrew Boie52fef672016-11-29 12:21:59 -08001189 if name not in self.extra_sections:
1190 recognized = False
Andrew Boie6acbe632015-07-17 12:03:52 -07001191
Anas Nashif3ba1d432017-12-05 15:28:44 -05001192 self.sections.append({"name": name, "load_addr": load_addr,
1193 "size": size, "virt_addr": virt_addr,
1194 "type": stype, "recognized": recognized})
Andrew Boie6acbe632015-07-17 12:03:52 -07001195
1196
Andrew Boie6acbe632015-07-17 12:03:52 -07001197# "list" - List of strings
1198# "list:<type>" - List of <type>
1199# "set" - Set of unordered, unique strings
1200# "set:<type>" - Set of <type>
1201# "float" - Floating point
1202# "int" - Integer
1203# "bool" - Boolean
1204# "str" - String
1205
1206# XXX Be sure to update __doc__ if you change any of this!!
1207
Anas Nashif83fc06a2019-06-22 11:04:10 -04001208platform_valid_keys = {
Anas Nashifd9882382019-12-12 09:58:28 -05001209 "supported_toolchains": {"type": "list", "default": []},
1210 "env": {"type": "list", "default": []}
1211}
Andrew Boie6acbe632015-07-17 12:03:52 -07001212
Anas Nashif3ba1d432017-12-05 15:28:44 -05001213testcase_valid_keys = {"tags": {"type": "set", "required": False},
1214 "type": {"type": "str", "default": "integration"},
1215 "extra_args": {"type": "list"},
1216 "extra_configs": {"type": "list"},
1217 "build_only": {"type": "bool", "default": False},
1218 "build_on_all": {"type": "bool", "default": False},
1219 "skip": {"type": "bool", "default": False},
1220 "slow": {"type": "bool", "default": False},
1221 "timeout": {"type": "int", "default": 60},
1222 "min_ram": {"type": "int", "default": 8},
1223 "depends_on": {"type": "set"},
1224 "min_flash": {"type": "int", "default": 32},
1225 "arch_whitelist": {"type": "set"},
1226 "arch_exclude": {"type": "set"},
1227 "extra_sections": {"type": "list", "default": []},
1228 "platform_exclude": {"type": "set"},
1229 "platform_whitelist": {"type": "set"},
1230 "toolchain_exclude": {"type": "set"},
1231 "toolchain_whitelist": {"type": "set"},
Anas Nashifab940162017-12-08 10:17:57 -05001232 "filter": {"type": "str"},
Anas Nashif576be982017-12-23 20:20:27 -05001233 "harness": {"type": "str"},
Praful Swarnakarcf89e282018-09-13 02:58:28 +05301234 "harness_config": {"type": "map", "default": {}}
Anas Nashifab940162017-12-08 10:17:57 -05001235 }
Andrew Boie6acbe632015-07-17 12:03:52 -07001236
Anas Nashif11ee5252019-12-04 12:59:10 -05001237
Andrew Boie6acbe632015-07-17 12:03:52 -07001238class SanityConfigParser:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001239 """Class to read test case files with semantic checking
Andrew Boie6acbe632015-07-17 12:03:52 -07001240 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001241
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001242 def __init__(self, filename, schema):
Andrew Boie6acbe632015-07-17 12:03:52 -07001243 """Instantiate a new SanityConfigParser object
1244
Anas Nashifa792a3d2017-04-04 18:47:49 -04001245 @param filename Source .yaml file to read
Andrew Boie6acbe632015-07-17 12:03:52 -07001246 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001247 self.data = {}
1248 self.schema = schema
Andrew Boie6acbe632015-07-17 12:03:52 -07001249 self.filename = filename
Anas Nashif255625b2017-12-05 15:08:26 -05001250 self.tests = {}
1251 self.common = {}
Anas Nashif83fc06a2019-06-22 11:04:10 -04001252
1253 def load(self):
1254 self.data = scl.yaml_load_verify(self.filename, self.schema)
1255
Anas Nashif255625b2017-12-05 15:08:26 -05001256 if 'tests' in self.data:
1257 self.tests = self.data['tests']
1258 if 'common' in self.data:
1259 self.common = self.data['common']
Andrew Boie6acbe632015-07-17 12:03:52 -07001260
Andrew Boie6acbe632015-07-17 12:03:52 -07001261 def _cast_value(self, value, typestr):
Anas Nashif3ba1d432017-12-05 15:28:44 -05001262 if isinstance(value, str):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001263 v = value.strip()
Andrew Boie6acbe632015-07-17 12:03:52 -07001264 if typestr == "str":
1265 return v
1266
1267 elif typestr == "float":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001268 return float(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001269
1270 elif typestr == "int":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001271 return int(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001272
1273 elif typestr == "bool":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001274 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001275
Anas Nashif3ba1d432017-12-05 15:28:44 -05001276 elif typestr.startswith("list") and isinstance(value, list):
Anas Nashiffa695d22017-10-04 16:14:27 -04001277 return value
Anas Nashif3ba1d432017-12-05 15:28:44 -05001278 elif typestr.startswith("list") and isinstance(value, str):
Andrew Boie6acbe632015-07-17 12:03:52 -07001279 vs = v.split()
1280 if len(typestr) > 4 and typestr[4] == ":":
1281 return [self._cast_value(vsi, typestr[5:]) for vsi in vs]
1282 else:
1283 return vs
1284
1285 elif typestr.startswith("set"):
1286 vs = v.split()
1287 if len(typestr) > 3 and typestr[3] == ":":
Anas Nashif83fc06a2019-06-22 11:04:10 -04001288 return {self._cast_value(vsi, typestr[4:]) for vsi in vs}
Andrew Boie6acbe632015-07-17 12:03:52 -07001289 else:
1290 return set(vs)
1291
Anas Nashif576be982017-12-23 20:20:27 -05001292 elif typestr.startswith("map"):
1293 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001294 else:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001295 raise ConfigurationError(
1296 self.filename, "unknown type '%s'" % value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001297
Anas Nashifb4754ed2017-12-05 17:27:58 -05001298 def get_test(self, name, valid_keys):
1299 """Get a dictionary representing the keys/values within a test
Andrew Boie6acbe632015-07-17 12:03:52 -07001300
Anas Nashifb4754ed2017-12-05 17:27:58 -05001301 @param name The test in the .yaml file to retrieve data from
Andrew Boie6acbe632015-07-17 12:03:52 -07001302 @param valid_keys A dictionary representing the intended semantics
Anas Nashifb4754ed2017-12-05 17:27:58 -05001303 for this test. Each key in this dictionary is a key that could
Anas Nashifa792a3d2017-04-04 18:47:49 -04001304 be specified, if a key is given in the .yaml file which isn't in
Andrew Boie6acbe632015-07-17 12:03:52 -07001305 here, it will generate an error. Each value in this dictionary
1306 is another dictionary containing metadata:
1307
1308 "default" - Default value if not given
1309 "type" - Data type to convert the text value to. Simple types
1310 supported are "str", "float", "int", "bool" which will get
1311 converted to respective Python data types. "set" and "list"
1312 may also be specified which will split the value by
1313 whitespace (but keep the elements as strings). finally,
1314 "list:<type>" and "set:<type>" may be given which will
1315 perform a type conversion after splitting the value up.
1316 "required" - If true, raise an error if not defined. If false
1317 and "default" isn't specified, a type conversion will be
1318 done on an empty string
Anas Nashifb4754ed2017-12-05 17:27:58 -05001319 @return A dictionary containing the test key-value pairs with
Andrew Boie6acbe632015-07-17 12:03:52 -07001320 type conversion and default values filled in per valid_keys
1321 """
1322
1323 d = {}
Anas Nashif255625b2017-12-05 15:08:26 -05001324 for k, v in self.common.items():
Anas Nashiffa695d22017-10-04 16:14:27 -04001325 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001326
Anas Nashifb4754ed2017-12-05 17:27:58 -05001327 for k, v in self.tests[name].items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001328 if k not in valid_keys:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001329 raise ConfigurationError(
1330 self.filename,
1331 "Unknown config key '%s' in definition for '%s'" %
Anas Nashifb4754ed2017-12-05 17:27:58 -05001332 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001333
Anas Nashiffa695d22017-10-04 16:14:27 -04001334 if k in d:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001335 if isinstance(d[k], str):
Paul Sokolovsky8c6d2e22019-08-29 13:20:57 +03001336 # By default, we just concatenate string values of keys
1337 # which appear both in "common" and per-test sections,
1338 # but some keys are handled in adhoc way based on their
1339 # semantics.
1340 if k == "filter":
1341 d[k] = "(%s) and (%s)" % (d[k], v)
1342 else:
1343 d[k] += " " + v
Anas Nashiffa695d22017-10-04 16:14:27 -04001344 else:
1345 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001346
Andrew Boie08ce5a52016-02-22 13:28:10 -08001347 for k, kinfo in valid_keys.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001348 if k not in d:
1349 if "required" in kinfo:
1350 required = kinfo["required"]
1351 else:
1352 required = False
1353
1354 if required:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001355 raise ConfigurationError(
1356 self.filename,
Anas Nashifb4754ed2017-12-05 17:27:58 -05001357 "missing required value for '%s' in test '%s'" %
1358 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001359 else:
1360 if "default" in kinfo:
1361 default = kinfo["default"]
1362 else:
1363 default = self._cast_value("", kinfo["type"])
1364 d[k] = default
1365 else:
1366 try:
1367 d[k] = self._cast_value(d[k], kinfo["type"])
Anas Nashif83fc06a2019-06-22 11:04:10 -04001368 except ValueError:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001369 raise ConfigurationError(
Anas Nashifb4754ed2017-12-05 17:27:58 -05001370 self.filename, "bad %s value '%s' for key '%s' in name '%s'" %
Anas Nashifd9882382019-12-12 09:58:28 -05001371 (kinfo["type"], d[k], k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001372
1373 return d
1374
1375
1376class Platform:
1377 """Class representing metadata for a particular platform
1378
Anas Nashifc7406082015-12-13 15:00:31 -05001379 Maps directly to BOARD when building"""
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001380
Anas Nashif83fc06a2019-06-22 11:04:10 -04001381 platform_schema = scl.yaml_load(os.path.join(ZEPHYR_BASE,
Anas Nashifd9882382019-12-12 09:58:28 -05001382 "scripts", "sanity_chk", "platform-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001383
Anas Nashif83fc06a2019-06-22 11:04:10 -04001384 def __init__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001385 """Constructor.
1386
Andrew Boie6acbe632015-07-17 12:03:52 -07001387 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001388
1389 self.name = ""
1390 self.sanitycheck = True
1391 # if no RAM size is specified by the board, take a default of 128K
1392 self.ram = 128
1393
1394 self.ignore_tags = []
1395 self.default = False
1396 # if no flash size is specified by the board, take a default of 512K
1397 self.flash = 512
1398 self.supported = set()
1399
1400 self.arch = ""
1401 self.type = "na"
1402 self.simulation = "na"
1403 self.supported_toolchains = []
1404 self.env = []
1405 self.env_satisfied = True
1406 self.filter_data = dict()
1407
1408 def load(self, platform_file):
1409 scp = SanityConfigParser(platform_file, self.platform_schema)
1410 scp.load()
Anas Nashif255625b2017-12-05 15:08:26 -05001411 data = scp.data
Anas Nashifa792a3d2017-04-04 18:47:49 -04001412
Anas Nashif255625b2017-12-05 15:08:26 -05001413 self.name = data['identifier']
Anas Nashiff3d48e12018-07-24 08:14:42 -05001414 self.sanitycheck = data.get("sanitycheck", True)
Anas Nashifa792a3d2017-04-04 18:47:49 -04001415 # if no RAM size is specified by the board, take a default of 128K
Anas Nashif255625b2017-12-05 15:08:26 -05001416 self.ram = data.get("ram", 128)
1417 testing = data.get("testing", {})
Anas Nashifa792a3d2017-04-04 18:47:49 -04001418 self.ignore_tags = testing.get("ignore_tags", [])
1419 self.default = testing.get("default", False)
1420 # if no flash size is specified by the board, take a default of 512K
Anas Nashif255625b2017-12-05 15:08:26 -05001421 self.flash = data.get("flash", 512)
Anas Nashif95276932017-07-31 08:47:41 -04001422 self.supported = set()
Anas Nashif255625b2017-12-05 15:08:26 -05001423 for supp_feature in data.get("supported", []):
Anas Nashif95276932017-07-31 08:47:41 -04001424 for item in supp_feature.split(":"):
1425 self.supported.add(item)
1426
Anas Nashif255625b2017-12-05 15:08:26 -05001427 self.arch = data['arch']
Anas Nashifcc164222017-12-26 11:02:46 -05001428 self.type = data.get('type', "na")
Anas Nashif8acdbd72018-01-04 14:15:22 -05001429 self.simulation = data.get('simulation', "na")
Anas Nashif255625b2017-12-05 15:08:26 -05001430 self.supported_toolchains = data.get("toolchain", [])
Anas Nashif924a4e72018-10-18 12:25:55 -04001431 self.env = data.get("env", [])
1432 self.env_satisfied = True
1433 for env in self.env:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001434 if not os.environ.get(env, None):
Anas Nashif924a4e72018-10-18 12:25:55 -04001435 self.env_satisfied = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001436
Andrew Boie6acbe632015-07-17 12:03:52 -07001437 def __repr__(self):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001438 return "<%s on %s>" % (self.name, self.arch)
Andrew Boie6acbe632015-07-17 12:03:52 -07001439
Anas Nashif11ee5252019-12-04 12:59:10 -05001440
Anas Nashif83fc06a2019-06-22 11:04:10 -04001441class TestCase(object):
Andrew Boie6acbe632015-07-17 12:03:52 -07001442 """Class representing a test application
1443 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001444
Anas Nashif83fc06a2019-06-22 11:04:10 -04001445 def __init__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001446 """TestCase constructor.
1447
Anas Nashif877d3ca2017-12-05 17:39:29 -05001448 This gets called by TestSuite as it finds and reads test yaml files.
Anas Nashifa792a3d2017-04-04 18:47:49 -04001449 Multiple TestCase instances may be generated from a single testcase.yaml,
Anas Nashif877d3ca2017-12-05 17:39:29 -05001450 each one corresponds to an entry within that file.
Andrew Boie6acbe632015-07-17 12:03:52 -07001451
Andrew Boie6acbe632015-07-17 12:03:52 -07001452 We need to have a unique name for every single test case. Since
Anas Nashifa792a3d2017-04-04 18:47:49 -04001453 a testcase.yaml can define multiple tests, the canonical name for
Andrew Boie6acbe632015-07-17 12:03:52 -07001454 the test case is <workdir>/<name>.
1455
Marc Herbert1c8632c2019-04-15 17:58:45 -07001456 @param testcase_root os.path.abspath() of one of the --testcase-root
1457 @param workdir Sub-directory of testcase_root where the
1458 .yaml test configuration file was found
Anas Nashif877d3ca2017-12-05 17:39:29 -05001459 @param name Name of this test case, corresponding to the entry name
Andrew Boie6acbe632015-07-17 12:03:52 -07001460 in the test case configuration file. For many test cases that just
1461 define one test, can be anything and is usually "test". This is
1462 really only used to distinguish between different cases when
Anas Nashifa792a3d2017-04-04 18:47:49 -04001463 the testcase.yaml defines multiple tests
Anas Nashif877d3ca2017-12-05 17:39:29 -05001464 @param tc_dict Dictionary with test values for this test case
Anas Nashifa792a3d2017-04-04 18:47:49 -04001465 from the testcase.yaml file
Andrew Boie6acbe632015-07-17 12:03:52 -07001466 """
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001467
Anas Nashif83fc06a2019-06-22 11:04:10 -04001468 self.id = ""
1469 self.source_dir = ""
1470 self.yamlfile = ""
Anas Nashifaae71d72018-04-21 22:26:48 -05001471 self.cases = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04001472 self.name = ""
Anas Nashifbd166f42017-09-02 12:32:08 -04001473
Anas Nashif83fc06a2019-06-22 11:04:10 -04001474 self.type = None
1475 self.tags = None
1476 self.extra_args = None
1477 self.extra_configs = None
1478 self.arch_whitelist = None
1479 self.arch_exclude = None
1480 self.skip = None
1481 self.platform_exclude = None
1482 self.platform_whitelist = None
1483 self.toolchain_exclude = None
1484 self.toolchain_whitelist = None
1485 self.tc_filter = None
1486 self.timeout = 60
1487 self.harness = ""
1488 self.harness_config = {}
1489 self.build_only = True
1490 self.build_on_all = False
1491 self.slow = False
1492 self.min_ram = None
1493 self.depends_on = None
1494 self.min_flash = None
1495 self.extra_sections = None
Andrew Boie6acbe632015-07-17 12:03:52 -07001496
Anas Nashif83fc06a2019-06-22 11:04:10 -04001497 @staticmethod
1498 def get_unique(testcase_root, workdir, name):
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001499
Marc Herbert1c8632c2019-04-15 17:58:45 -07001500 canonical_testcase_root = os.path.realpath(testcase_root)
Tyler Hallbb359252019-06-19 02:11:35 -04001501 if Path(canonical_zephyr_base) in Path(canonical_testcase_root).parents:
Marc Herbert1c8632c2019-04-15 17:58:45 -07001502 # This is in ZEPHYR_BASE, so include path in name for uniqueness
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001503 # FIXME: We should not depend on path of test for unique names.
Marc Herbert1c8632c2019-04-15 17:58:45 -07001504 relative_tc_root = os.path.relpath(canonical_testcase_root,
Anas Nashif11ee5252019-12-04 12:59:10 -05001505 start=canonical_zephyr_base)
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001506 else:
Marc Herbert1c8632c2019-04-15 17:58:45 -07001507 relative_tc_root = ""
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001508
Marc Herbert1c8632c2019-04-15 17:58:45 -07001509 # workdir can be "."
1510 unique = os.path.normpath(os.path.join(relative_tc_root, workdir, name))
Anas Nashiff18ad9d2018-11-20 09:03:17 -05001511 return unique
1512
Anas Nashif83fc06a2019-06-22 11:04:10 -04001513 @staticmethod
1514 def scan_file(inf_name):
Anas Nashifaae71d72018-04-21 22:26:48 -05001515 suite_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001516 # do not match until end-of-line, otherwise we won't allow
1517 # stc_regex below to catch the ones that are declared in the same
1518 # line--as we only search starting the end of this match
1519 br"^\s*ztest_test_suite\(\s*(?P<suite_name>[a-zA-Z0-9_]+)\s*,",
Anas Nashifaae71d72018-04-21 22:26:48 -05001520 re.MULTILINE)
1521 stc_regex = re.compile(
Anas Nashifd9882382019-12-12 09:58:28 -05001522 br"^\s*" # empy space at the beginning is ok
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001523 # catch the case where it is declared in the same sentence, e.g:
1524 #
1525 # ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
1526 br"(?:ztest_test_suite\([a-zA-Z0-9_]+,\s*)?"
1527 # Catch ztest[_user]_unit_test-[_setup_teardown](TESTNAME)
Anas Nashif9091a012019-11-24 09:22:22 -05001528 br"ztest_(?:1cpu_)?(?:user_)?unit_test(?:_setup_teardown)?"
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001529 # Consume the argument that becomes the extra testcse
1530 br"\(\s*"
1531 br"(?P<stc_name>[a-zA-Z0-9_]+)"
1532 # _setup_teardown() variant has two extra arguments that we ignore
1533 br"(?:\s*,\s*[a-zA-Z0-9_]+\s*,\s*[a-zA-Z0-9_]+)?"
1534 br"\s*\)",
1535 # We don't check how it finishes; we don't care
Anas Nashifaae71d72018-04-21 22:26:48 -05001536 re.MULTILINE)
1537 suite_run_regex = re.compile(
1538 br"^\s*ztest_run_test_suite\((?P<suite_name>[a-zA-Z0-9_]+)\)",
1539 re.MULTILINE)
1540 achtung_regex = re.compile(
1541 br"(#ifdef|#endif)",
1542 re.MULTILINE)
1543 warnings = None
1544
1545 with open(inf_name) as inf:
Anas Nashif19d67e42019-11-21 11:33:12 -05001546 if os.name == 'nt':
Anas Nashifd9882382019-12-12 09:58:28 -05001547 mmap_args = {'fileno': inf.fileno(), 'length': 0, 'access': mmap.ACCESS_READ}
Anas Nashif19d67e42019-11-21 11:33:12 -05001548 else:
Anas Nashifd9882382019-12-12 09:58:28 -05001549 mmap_args = {'fileno': inf.fileno(), 'length': 0, 'flags': mmap.MAP_PRIVATE, 'prot': mmap.PROT_READ,
1550 'offset': 0}
Anas Nashif19d67e42019-11-21 11:33:12 -05001551
1552 with contextlib.closing(mmap.mmap(**mmap_args)) as main_c:
Ulf Magnusson5c2e8142019-10-29 11:50:26 +01001553 # contextlib makes pylint think main_c isn't subscriptable
1554 # pylint: disable=unsubscriptable-object
1555
Anas Nashifaae71d72018-04-21 22:26:48 -05001556 suite_regex_match = suite_regex.search(main_c)
1557 if not suite_regex_match:
1558 # can't find ztest_test_suite, maybe a client, because
1559 # it includes ztest.h
1560 return None, None
1561
1562 suite_run_match = suite_run_regex.search(main_c)
1563 if not suite_run_match:
1564 raise ValueError("can't find ztest_run_test_suite")
1565
1566 achtung_matches = re.findall(
1567 achtung_regex,
1568 main_c[suite_regex_match.end():suite_run_match.start()])
1569 if achtung_matches:
1570 warnings = "found invalid %s in ztest_test_suite()" \
Anas Nashif83fc06a2019-06-22 11:04:10 -04001571 % ", ".join({match.decode() for match in achtung_matches})
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001572 _matches = re.findall(
Anas Nashifaae71d72018-04-21 22:26:48 -05001573 stc_regex,
1574 main_c[suite_regex_match.end():suite_run_match.start()])
Anas Nashifd9882382019-12-12 09:58:28 -05001575 matches = [match.decode().replace("test_", "") for match in _matches]
Anas Nashifaae71d72018-04-21 22:26:48 -05001576 return matches, warnings
1577
1578 def scan_path(self, path):
1579 subcases = []
1580 for filename in glob.glob(os.path.join(path, "src", "*.c")):
1581 try:
1582 _subcases, warnings = self.scan_file(filename)
1583 if warnings:
Anas Nashif7a361b82019-12-06 11:37:40 -05001584 logger.error("%s: %s" % (filename, warnings))
Anas Nashifaae71d72018-04-21 22:26:48 -05001585 if _subcases:
1586 subcases += _subcases
1587 except ValueError as e:
Anas Nashif7a361b82019-12-06 11:37:40 -05001588 logger.error("%s: can't find: %s" % (filename, e))
Anas Nashif434995c2019-12-01 13:55:11 -05001589 for filename in glob.glob(os.path.join(path, "*.c")):
1590 try:
1591 _subcases, warnings = self.scan_file(filename)
1592 if warnings:
Anas Nashif7a361b82019-12-06 11:37:40 -05001593 logger.error("%s: %s" % (filename, warnings))
Anas Nashif434995c2019-12-01 13:55:11 -05001594 if _subcases:
1595 subcases += _subcases
1596 except ValueError as e:
Anas Nashif7a361b82019-12-06 11:37:40 -05001597 logger.error("%s: can't find: %s" % (filename, e))
Anas Nashifaae71d72018-04-21 22:26:48 -05001598 return subcases
1599
Anas Nashif83fc06a2019-06-22 11:04:10 -04001600 def parse_subcases(self, test_path):
Anas Nashif9091a012019-11-24 09:22:22 -05001601 results = self.scan_path(test_path)
Anas Nashifaae71d72018-04-21 22:26:48 -05001602 for sub in results:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001603 name = "{}.{}".format(self.id, sub)
Anas Nashifaae71d72018-04-21 22:26:48 -05001604 self.cases.append(name)
1605
Anas Nashiff16e92c2019-03-31 16:58:12 -04001606 if not results:
1607 self.cases.append(self.id)
1608
Anas Nashif75547e22018-02-24 08:32:14 -06001609 def __str__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001610 return self.name
1611
1612
Andrew Boie6acbe632015-07-17 12:03:52 -07001613class TestInstance:
1614 """Class representing the execution of a particular TestCase on a platform
1615
1616 @param test The TestCase object we want to build/execute
1617 @param platform Platform object that we want to build and run against
1618 @param base_outdir Base directory for all test results. The actual
1619 out directory used is <outdir>/<platform>/<test case name>
1620 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001621
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05001622 def __init__(self, testcase, platform, outdir):
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001623
Anas Nashif83fc06a2019-06-22 11:04:10 -04001624 self.testcase = testcase
1625 self.platform = platform
1626
1627 self.status = None
1628 self.reason = "N/A"
1629 self.metrics = dict()
1630 self.handler = None
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05001631 self.outdir = outdir
Anas Nashif83fc06a2019-06-22 11:04:10 -04001632
1633 self.name = os.path.join(platform.name, testcase.name)
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05001634 self.build_dir = os.path.join(outdir, platform.name, testcase.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04001635
Anas Nashif56656842019-12-10 12:26:00 -05001636 self.build_only = True
1637 self.run = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001638
Anas Nashife0a6a0b2018-02-15 20:07:24 -06001639 self.results = {}
Andrew Boie6acbe632015-07-17 12:03:52 -07001640
Marc Herbert0f7255c2019-04-05 14:14:21 -07001641 def __lt__(self, other):
1642 return self.name < other.name
1643
Anas Nashif56656842019-12-10 12:26:00 -05001644 def check_build_or_run(self, build_only=False, enable_slow=False, device_testing=False, fixture=[]):
1645
Anas Nashif19d67e42019-11-21 11:33:12 -05001646 # right now we only support building on windows. running is still work
1647 # in progress.
Anas Nashif19d67e42019-11-21 11:33:12 -05001648 if os.name == 'nt':
Anas Nashif56656842019-12-10 12:26:00 -05001649 self.build_only = True
1650 self.run = False
1651 return
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001652
Anas Nashif56656842019-12-10 12:26:00 -05001653 _build_only = True
Anas Nashif83fc06a2019-06-22 11:04:10 -04001654
1655 # we asked for build-only on the command line
Anas Nashif56656842019-12-10 12:26:00 -05001656 if build_only or self.testcase.build_only:
1657 self.build_only = True
1658 self.run = False
1659 return
Anas Nashif83fc06a2019-06-22 11:04:10 -04001660
1661 # Do not run slow tests:
Anas Nashif56656842019-12-10 12:26:00 -05001662 skip_slow = self.testcase.slow and not enable_slow
Anas Nashif83fc06a2019-06-22 11:04:10 -04001663 if skip_slow:
Anas Nashif56656842019-12-10 12:26:00 -05001664 self.build_only = True
1665 self.run = False
1666 return
Anas Nashif83fc06a2019-06-22 11:04:10 -04001667
Anas Nashifd9882382019-12-12 09:58:28 -05001668 runnable = bool(self.testcase.type == "unit" or \
1669 self.platform.type == "native" or \
1670 self.platform.simulation in ["nsim", "renode", "qemu"] or \
1671 device_testing)
Anas Nashif83fc06a2019-06-22 11:04:10 -04001672
1673 if self.platform.simulation == "nsim":
1674 if not find_executable("nsimdrv"):
1675 runnable = False
1676
1677 if self.platform.simulation == "renode":
1678 if not find_executable("renode"):
1679 runnable = False
1680
1681 # console harness allows us to run the test and capture data.
1682 if self.testcase.harness == 'console':
1683
1684 # if we have a fixture that is also being supplied on the
1685 # command-line, then we need to run the test, not just build it.
1686 if "fixture" in self.testcase.harness_config:
Anas Nashifd9882382019-12-12 09:58:28 -05001687 fixture_cfg = self.testcase.harness_config['fixture']
1688 if fixture_cfg in fixture:
Anas Nashif56656842019-12-10 12:26:00 -05001689 _build_only = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001690 else:
Anas Nashif56656842019-12-10 12:26:00 -05001691 _build_only = True
Anas Nashif83fc06a2019-06-22 11:04:10 -04001692 else:
Anas Nashif56656842019-12-10 12:26:00 -05001693 _build_only = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001694 elif self.testcase.harness:
Anas Nashif56656842019-12-10 12:26:00 -05001695 _build_only = True
Anas Nashif83fc06a2019-06-22 11:04:10 -04001696 else:
Anas Nashif56656842019-12-10 12:26:00 -05001697 _build_only = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04001698
Anas Nashif56656842019-12-10 12:26:00 -05001699 self.build_only = not (not _build_only and runnable)
1700 self.run = not self.build_only
1701 return
Anas Nashif7dd19ea2018-11-14 08:46:49 -05001702
Anas Nashif56656842019-12-10 12:26:00 -05001703 def create_overlay(self, platform, enable_asan=False, enable_coverage=False, coverage_platform=[]):
Marc Herbertc7633de2019-07-06 15:52:31 -07001704 # Create this in a "sanitycheck/" subdirectory otherwise this
1705 # will pass this overlay to kconfig.py *twice* and kconfig.cmake
1706 # will silently give that second time precedence over any
1707 # --extra-args=CONFIG_*
Anas Nashif83fc06a2019-06-22 11:04:10 -04001708 subdir = os.path.join(self.build_dir, "sanitycheck")
Marc Herbertc7633de2019-07-06 15:52:31 -07001709 os.makedirs(subdir, exist_ok=True)
1710 file = os.path.join(subdir, "testcase_extra.conf")
Anas Nashif56656842019-12-10 12:26:00 -05001711
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001712 with open(file, "w") as f:
1713 content = ""
Anas Nashif3cbffef2018-11-07 23:50:54 -05001714
Anas Nashif83fc06a2019-06-22 11:04:10 -04001715 if self.testcase.extra_configs:
1716 content = "\n".join(self.testcase.extra_configs)
Anas Nashif3cbffef2018-11-07 23:50:54 -05001717
Anas Nashif56656842019-12-10 12:26:00 -05001718 if enable_coverage:
1719 if platform.name in coverage_platform:
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001720 content = content + "\nCONFIG_COVERAGE=y"
1721
Anas Nashif56656842019-12-10 12:26:00 -05001722 if enable_asan:
Jan Van Winkel21212f32019-09-12 00:03:35 +02001723 if platform.type == "native":
1724 content = content + "\nCONFIG_ASAN=y"
1725
Anas Nashiff4d8ecc2019-03-02 15:43:23 -05001726 f.write(content)
Anas Nashiffa695d22017-10-04 16:14:27 -04001727
Andrew Boie6acbe632015-07-17 12:03:52 -07001728 def calculate_sizes(self):
1729 """Get the RAM/ROM sizes of a test case.
1730
1731 This can only be run after the instance has been executed by
1732 MakeGenerator, otherwise there won't be any binaries to measure.
1733
1734 @return A SizeCalculator object
1735 """
Anas Nashif83fc06a2019-06-22 11:04:10 -04001736 fns = glob.glob(os.path.join(self.build_dir, "zephyr", "*.elf"))
1737 fns.extend(glob.glob(os.path.join(self.build_dir, "zephyr", "*.exe")))
Anas Nashiff8dcad42016-10-27 18:10:08 -04001738 fns = [x for x in fns if not x.endswith('_prebuilt.elf')]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001739 if len(fns) != 1:
Andrew Boie5d4eb782015-10-02 10:04:56 -07001740 raise BuildError("Missing/multiple output ELF binary")
Anas Nashif83fc06a2019-06-22 11:04:10 -04001741
1742 return SizeCalculator(fns[0], self.testcase.extra_sections)
Andrew Boie6acbe632015-07-17 12:03:52 -07001743
1744 def __repr__(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -04001745 return "<TestCase %s on %s>" % (self.testcase.name, self.platform.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07001746
1747
Anas Nashif83fc06a2019-06-22 11:04:10 -04001748class CMake():
Anas Nashif83fc06a2019-06-22 11:04:10 -04001749 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
1750 dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
1751
1752 def __init__(self, testcase, platform, source_dir, build_dir):
1753
1754 self.cwd = None
1755 self.capture_output = True
1756
1757 self.defconfig = {}
1758 self.cmake_cache = {}
Anas Nashif83fc06a2019-06-22 11:04:10 -04001759
1760 self.instance = None
1761 self.testcase = testcase
1762 self.platform = platform
1763 self.source_dir = source_dir
1764 self.build_dir = build_dir
1765 self.log = "build.log"
1766
1767 def parse_generated(self):
1768 self.defconfig = {}
1769 return {}
1770
1771 def run_build(self, args=[]):
1772
Anas Nashif7a361b82019-12-06 11:37:40 -05001773 logger.debug("Building %s for %s" % (self.source_dir, self.platform.name))
Anas Nashif83fc06a2019-06-22 11:04:10 -04001774
1775 cmake_args = []
1776 cmake_args.extend(args)
1777 cmake = shutil.which('cmake')
1778 cmd = [cmake] + cmake_args
1779 kwargs = dict()
1780
1781 if self.capture_output:
1782 kwargs['stdout'] = subprocess.PIPE
1783 # CMake sends the output of message() to stderr unless it's STATUS
1784 kwargs['stderr'] = subprocess.STDOUT
1785
1786 if self.cwd:
1787 kwargs['cwd'] = self.cwd
1788
1789 p = subprocess.Popen(cmd, **kwargs)
1790 out, _ = p.communicate()
1791
1792 results = {}
1793 if p.returncode == 0:
Anas Nashifd9882382019-12-12 09:58:28 -05001794 msg = "Finished building %s for %s" % (self.source_dir, self.platform.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04001795
1796 self.instance.status = "passed"
Anas Nashif83fc06a2019-06-22 11:04:10 -04001797 results = {'msg': msg, "returncode": p.returncode, "instance": self.instance}
1798
1799 if out:
1800 log_msg = out.decode(sys.getdefaultencoding())
1801 with open(os.path.join(self.build_dir, self.log), "a") as log:
1802 log.write(log_msg)
1803
1804 else:
1805 return None
1806 else:
1807 # A real error occurred, raise an exception
1808 if out:
1809 log_msg = out.decode(sys.getdefaultencoding())
1810 with open(os.path.join(self.build_dir, self.log), "a") as log:
1811 log.write(log_msg)
1812
1813 overflow_flash = "region `FLASH' overflowed by"
1814 overflow_ram = "region `RAM' overflowed by"
1815
1816 if log_msg:
1817 if log_msg.find(overflow_flash) > 0 or log_msg.find(overflow_ram) > 0:
Anas Nashif7a361b82019-12-06 11:37:40 -05001818 logger.debug("RAM/ROM Overflow")
Anas Nashif83fc06a2019-06-22 11:04:10 -04001819 self.instance.status = "skipped"
1820 self.instance.reason = "overflow"
1821 else:
1822 self.instance.status = "failed"
1823 self.instance.reason = "Build failure"
1824
1825 results = {
Anas Nashifd9882382019-12-12 09:58:28 -05001826 "returncode": p.returncode,
1827 "instance": self.instance,
1828 }
Anas Nashif83fc06a2019-06-22 11:04:10 -04001829
1830 return results
1831
1832 def run_cmake(self, args=[]):
1833
Anas Nashif11ee5252019-12-04 12:59:10 -05001834 ldflags = "-Wl,--fatal-warnings"
Anas Nashifd9882382019-12-12 09:58:28 -05001835 logger.debug("Running cmake on %s for %s" % (self.source_dir, self.platform.name))
Anas Nashif83fc06a2019-06-22 11:04:10 -04001836
Anas Nashif11ee5252019-12-04 12:59:10 -05001837 # fixme: add additional cflags based on options
Anas Nashifa5984ab2019-10-22 07:36:24 -07001838 cmake_args = [
Anas Nashifd9882382019-12-12 09:58:28 -05001839 '-B{}'.format(self.build_dir),
1840 '-S{}'.format(self.source_dir),
1841 '-DEXTRA_CFLAGS="-Werror ',
1842 '-DEXTRA_AFLAGS=-Wa,--fatal-warnings',
1843 '-DEXTRA_LDFLAGS="{}'.format(ldflags),
1844 '-G{}'.format(get_generator()[1])
1845 ]
Anas Nashif83fc06a2019-06-22 11:04:10 -04001846
Anas Nashifd91f9932019-11-30 10:15:23 -05001847 if options.cmake_only:
1848 cmake_args.append("-DCMAKE_EXPORT_COMPILE_COMMANDS=1")
1849
Anas Nashif83fc06a2019-06-22 11:04:10 -04001850 args = ["-D{}".format(a.replace('"', '')) for a in args]
1851 cmake_args.extend(args)
1852
1853 cmake_opts = ['-DBOARD={}'.format(self.platform.name)]
1854 cmake_args.extend(cmake_opts)
1855
1856 cmake = shutil.which('cmake')
1857 cmd = [cmake] + cmake_args
1858 kwargs = dict()
1859
1860 if self.capture_output:
1861 kwargs['stdout'] = subprocess.PIPE
1862 # CMake sends the output of message() to stderr unless it's STATUS
1863 kwargs['stderr'] = subprocess.STDOUT
1864
1865 if self.cwd:
1866 kwargs['cwd'] = self.cwd
1867
1868 p = subprocess.Popen(cmd, **kwargs)
1869 out, _ = p.communicate()
1870
1871 if p.returncode == 0:
1872 filter_results = self.parse_generated()
Anas Nashifd9882382019-12-12 09:58:28 -05001873 msg = "Finished building %s for %s" % (self.source_dir, self.platform.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04001874
1875 results = {'msg': msg, 'filter': filter_results}
1876
1877 else:
1878 self.instance.status = "failed"
1879 self.instance.reason = "Cmake build failure"
1880 results = {"returncode": p.returncode}
1881
Anas Nashif83fc06a2019-06-22 11:04:10 -04001882 if out:
1883 with open(os.path.join(self.build_dir, self.log), "a") as log:
1884 log_msg = out.decode(sys.getdefaultencoding())
1885 log.write(log_msg)
1886
1887 return results
1888
1889
1890class FilterBuilder(CMake):
1891
1892 def __init__(self, testcase, platform, source_dir, build_dir):
1893 super().__init__(testcase, platform, source_dir, build_dir)
1894
1895 self.log = "config-sanitycheck.log"
1896
1897 def parse_generated(self):
1898
1899 if self.platform.name == "unit_testing":
1900 return {}
1901
Anas Nashif83fc06a2019-06-22 11:04:10 -04001902 cmake_cache_path = os.path.join(self.build_dir, "CMakeCache.txt")
Anas Nashif83fc06a2019-06-22 11:04:10 -04001903 defconfig_path = os.path.join(self.build_dir, "zephyr", ".config")
1904
1905 with open(defconfig_path, "r") as fp:
1906 defconfig = {}
1907 for line in fp.readlines():
1908 m = self.config_re.match(line)
1909 if not m:
1910 if line.strip() and not line.startswith("#"):
1911 sys.stderr.write("Unrecognized line %s\n" % line)
1912 continue
1913 defconfig[m.group(1)] = m.group(2).strip()
1914
1915 self.defconfig = defconfig
1916
1917 cmake_conf = {}
1918 try:
1919 cache = CMakeCache.from_file(cmake_cache_path)
1920 except FileNotFoundError:
1921 cache = {}
1922
1923 for k in iter(cache):
1924 cmake_conf[k.name] = k.value
1925
1926 self.cmake_cache = cmake_conf
1927
Anas Nashif83fc06a2019-06-22 11:04:10 -04001928 filter_data = {
1929 "ARCH": self.platform.arch,
1930 "PLATFORM": self.platform.name
1931 }
1932 filter_data.update(os.environ)
1933 filter_data.update(self.defconfig)
1934 filter_data.update(self.cmake_cache)
Anas Nashif83fc06a2019-06-22 11:04:10 -04001935
Anas Nashif556f3cb2019-11-05 15:36:15 -08001936 dts_path = os.path.join(self.build_dir, "zephyr", self.platform.name + ".dts.pre.tmp")
Kumar Galaae88e442019-11-06 04:55:24 -06001937 if self.testcase and self.testcase.tc_filter:
Anas Nashif83fc06a2019-06-22 11:04:10 -04001938 try:
Kumar Galaae88e442019-11-06 04:55:24 -06001939 if os.path.exists(dts_path):
1940 edt = edtlib.EDT(dts_path, [os.path.join(ZEPHYR_BASE, "dts", "bindings")])
1941 else:
1942 edt = None
Kumar Gala7733b942019-09-12 17:08:43 -05001943 res = expr_parser.parse(self.testcase.tc_filter, filter_data, edt)
1944
Anas Nashif83fc06a2019-06-22 11:04:10 -04001945 except (ValueError, SyntaxError) as se:
1946 sys.stderr.write(
1947 "Failed processing %s\n" % self.testcase.yamlfile)
1948 raise se
1949
1950 if not res:
1951 return {os.path.join(self.platform.name, self.testcase.name): True}
1952 else:
1953 return {os.path.join(self.platform.name, self.testcase.name): False}
1954 else:
1955 self.platform.filter_data = filter_data
1956 return filter_data
1957
1958
1959class ProjectBuilder(FilterBuilder):
1960
Anas Nashif56656842019-12-10 12:26:00 -05001961 def __init__(self, suite, instance, **kwargs):
Anas Nashif83fc06a2019-06-22 11:04:10 -04001962 super().__init__(instance.testcase, instance.platform, instance.testcase.source_dir, instance.build_dir)
1963
1964 self.log = "build.log"
1965 self.instance = instance
1966 self.suite = suite
1967
Anas Nashif56656842019-12-10 12:26:00 -05001968 self.lsan = kwargs.get('lsan', False)
1969 self.asan = kwargs.get('asan', False)
1970 self.valgrind = kwargs.get('valgrind', False)
1971 self.extra_args = kwargs.get('extra_args', [])
1972 self.device_testing = kwargs.get('device_testing', False)
1973 self.cmake_only = kwargs.get('cmake_only', False)
1974 self.coverage = kwargs.get('coverage', False)
Anas Nashife9eb0092019-12-10 16:31:22 -05001975 self.inline_logs = kwargs.get('inline_logs', False)
1976
Anas Nashifd9882382019-12-12 09:58:28 -05001977 @staticmethod
1978 def log_info(filename, inline_logs):
Anas Nashife9eb0092019-12-10 16:31:22 -05001979 filename = os.path.relpath(os.path.realpath(filename))
1980 if inline_logs:
1981 logger.info("{:-^100}".format(filename))
1982
1983 try:
1984 with open(filename) as fp:
1985 data = fp.read()
1986 except Exception as e:
1987 data = "Unable to read log data (%s)\n" % (str(e))
1988
1989 logger.error(data)
1990
1991 logger.info("{:-^100}".format(filename))
1992 else:
1993 logger.error("see: " + COLOR_YELLOW + filename + COLOR_NORMAL)
1994
1995 def log_info_file(self, inline_logs):
1996 build_dir = self.instance.build_dir
1997 h_log = "{}/handler.log".format(build_dir)
1998 b_log = "{}/build.log".format(build_dir)
1999 v_log = "{}/valgrind.log".format(build_dir)
2000 d_log = "{}/device.log".format(build_dir)
2001
2002 if os.path.exists(v_log) and "Valgrind" in self.instance.reason:
2003 self.log_info("{}".format(v_log), inline_logs)
2004 elif os.path.exists(d_log):
2005 self.log_info("{}".format(d_log), inline_logs)
2006 elif os.path.exists(h_log):
2007 self.log_info("{}".format(h_log), inline_logs)
2008 else:
2009 self.log_info("{}".format(b_log), inline_logs)
Anas Nashif56656842019-12-10 12:26:00 -05002010
Anas Nashif83fc06a2019-06-22 11:04:10 -04002011 def setup_handler(self):
2012
2013 instance = self.instance
2014 args = []
2015
2016 # FIXME: Needs simplification
2017 if instance.platform.simulation == "qemu":
2018 instance.handler = QEMUHandler(instance, "qemu")
2019 args.append("QEMU_PIPE=%s" % instance.handler.get_fifo())
2020 instance.handler.call_make_run = True
2021 elif instance.testcase.type == "unit":
2022 instance.handler = BinaryHandler(instance, "unit")
2023 instance.handler.binary = os.path.join(instance.build_dir, "testbinary")
2024 elif instance.platform.type == "native":
Anas Nashif6c0e1702019-12-05 15:24:52 -05002025 handler = BinaryHandler(instance, "native")
2026
Anas Nashif56656842019-12-10 12:26:00 -05002027 handler.asan = self.asan
2028 handler.valgrind = self.valgrind
2029 handler.lsan = self.lsan
2030 handler.coverage = self.coverage
Anas Nashif6c0e1702019-12-05 15:24:52 -05002031
2032 handler.binary = os.path.join(instance.build_dir, "zephyr", "zephyr.exe")
2033 instance.handler = handler
Anas Nashif83fc06a2019-06-22 11:04:10 -04002034 elif instance.platform.simulation == "nsim":
2035 if find_executable("nsimdrv"):
2036 instance.handler = BinaryHandler(instance, "nsim")
2037 instance.handler.call_make_run = True
2038 elif instance.platform.simulation == "renode":
2039 if find_executable("renode"):
2040 instance.handler = BinaryHandler(instance, "renode")
2041 instance.handler.pid_fn = os.path.join(instance.build_dir, "renode.pid")
2042 instance.handler.call_make_run = True
Anas Nashif56656842019-12-10 12:26:00 -05002043 elif self.device_testing:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002044 instance.handler = DeviceHandler(instance, "device")
2045
2046 if instance.handler:
2047 instance.handler.args = args
2048
2049 def process(self, message):
2050 op = message.get('op')
2051
2052 if not self.instance.handler:
2053 self.setup_handler()
2054
2055 # The build process, call cmake and build with configured generator
2056 if op == "cmake":
2057 results = self.cmake()
2058 if self.instance.status == "failed":
2059 pipeline.put({"op": "report", "test": self.instance})
Anas Nashif56656842019-12-10 12:26:00 -05002060 elif self.cmake_only:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002061 pipeline.put({"op": "report", "test": self.instance})
2062 else:
2063 if self.instance.name in results['filter'] and results['filter'][self.instance.name]:
Anas Nashif7a361b82019-12-06 11:37:40 -05002064 logger.debug("filtering %s" % self.instance.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002065 self.instance.status = "skipped"
2066 self.instance.reason = "filter"
2067 pipeline.put({"op": "report", "test": self.instance})
2068 else:
2069 pipeline.put({"op": "build", "test": self.instance})
2070
Anas Nashif83fc06a2019-06-22 11:04:10 -04002071 elif op == "build":
Anas Nashifd9882382019-12-12 09:58:28 -05002072 logger.debug("build test: %s" % self.instance.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002073 results = self.build()
2074
2075 if results.get('returncode', 1) > 0:
2076 pipeline.put({"op": "report", "test": self.instance})
2077 else:
2078 if self.instance.run:
2079 pipeline.put({"op": "run", "test": self.instance})
2080 else:
2081 pipeline.put({"op": "report", "test": self.instance})
2082 # Run the generated binary using one of the supported handlers
2083 elif op == "run":
Anas Nashifd9882382019-12-12 09:58:28 -05002084 logger.debug("run test: %s" % self.instance.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002085 self.run()
2086 self.instance.status, _ = self.instance.handler.get_state()
Anas Nashif83fc06a2019-06-22 11:04:10 -04002087 pipeline.put({
2088 "op": "report",
2089 "test": self.instance,
2090 "state": "executed",
2091 "status": self.instance.status,
Stephanos Ioannidis93889002019-11-11 20:31:03 +09002092 "reason": self.instance.reason}
Anas Nashifd9882382019-12-12 09:58:28 -05002093 )
Anas Nashif83fc06a2019-06-22 11:04:10 -04002094
2095 # Report results and output progress to screen
2096 elif op == "report":
Anas Nashifc1ea4522019-10-11 07:32:45 -07002097 with report_lock:
2098 self.report_out()
2099
Anas Nashif83fc06a2019-06-22 11:04:10 -04002100 def report_out(self):
2101 total_tests_width = len(str(self.suite.total_tests))
2102 self.suite.total_done += 1
2103 instance = self.instance
2104
2105 if instance.status in ["failed", "timeout"]:
2106 self.suite.total_failed += 1
2107 if VERBOSE or not TERMINAL:
2108 status = COLOR_RED + "FAILED " + COLOR_NORMAL + instance.reason
2109 else:
Anas Nashif67f9ecb2019-12-09 10:46:19 -05002110 print("")
2111 logger.error(
2112 "{:<25} {:<50} {}FAILED{}: {}".format(
Anas Nashif83fc06a2019-06-22 11:04:10 -04002113 instance.platform.name,
2114 instance.testcase.name,
2115 COLOR_RED,
2116 COLOR_NORMAL,
Anas Nashif7a361b82019-12-06 11:37:40 -05002117 instance.reason))
Anas Nashifc1ea4522019-10-11 07:32:45 -07002118 if not VERBOSE:
Anas Nashife9eb0092019-12-10 16:31:22 -05002119 self.log_info_file(self.inline_logs)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002120 elif instance.status == "skipped":
2121 self.suite.total_skipped += 1
2122 status = COLOR_YELLOW + "SKIPPED" + COLOR_NORMAL
Anas Nashif83fc06a2019-06-22 11:04:10 -04002123 else:
2124 status = COLOR_GREEN + "PASSED" + COLOR_NORMAL
2125
2126 if VERBOSE or not TERMINAL:
Anas Nashif56656842019-12-10 12:26:00 -05002127 if self.cmake_only:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002128 more_info = "cmake"
2129 elif instance.status == "skipped":
2130 more_info = instance.reason
2131 else:
2132 if instance.handler and instance.run:
2133 more_info = instance.handler.type_str
2134 htime = instance.handler.duration
2135 if htime:
2136 more_info += " {:.3f}s".format(htime)
2137 else:
2138 more_info = "build"
2139
Anas Nashif7a361b82019-12-06 11:37:40 -05002140 logger.info("{:>{}}/{} {:<25} {:<50} {} ({})".format(
Anas Nashif83fc06a2019-06-22 11:04:10 -04002141 self.suite.total_done, total_tests_width, self.suite.total_tests, instance.platform.name,
2142 instance.testcase.name, status, more_info))
2143
2144 if instance.status in ["failed", "timeout"]:
Anas Nashifd9882382019-12-12 09:58:28 -05002145 self.log_info_file(self.inline_logs)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002146 else:
Anas Nashif67f9ecb2019-12-09 10:46:19 -05002147 sys.stdout.write("\rINFO - Total complete: %s%4d/%4d%s %2d%% skipped: %s%4d%s, failed: %s%4d%s" % (
Anas Nashifd9882382019-12-12 09:58:28 -05002148 COLOR_GREEN,
2149 self.suite.total_done,
2150 self.suite.total_tests,
2151 COLOR_NORMAL,
2152 int((float(self.suite.total_done) / self.suite.total_tests) * 100),
2153 COLOR_YELLOW if self.suite.total_skipped > 0 else COLOR_NORMAL,
2154 self.suite.total_skipped,
2155 COLOR_NORMAL,
2156 COLOR_RED if self.suite.total_failed > 0 else COLOR_NORMAL,
2157 self.suite.total_failed,
2158 COLOR_NORMAL
2159 )
2160 )
Anas Nashif83fc06a2019-06-22 11:04:10 -04002161 sys.stdout.flush()
2162
2163 def cmake(self):
2164
2165 instance = self.instance
2166 args = self.testcase.extra_args[:]
Anas Nashif56656842019-12-10 12:26:00 -05002167 args += self.extra_args
Anas Nashif83fc06a2019-06-22 11:04:10 -04002168
2169 if instance.handler:
2170 args += instance.handler.args
2171
2172 # merge overlay files into one variable
2173 overlays = ""
2174 idx = 0
2175 for arg in args:
2176 match = re.search('OVERLAY_CONFIG="(.*)"', arg)
2177 if match:
2178 overlays += match.group(1)
2179 del args[idx]
2180 idx += 1
2181
Anas Nashif56656842019-12-10 12:26:00 -05002182 if (self.testcase.extra_configs or self.coverage or
2183 self.asan):
Anas Nashifd9882382019-12-12 09:58:28 -05002184 args.append("OVERLAY_CONFIG=\"%s %s\"" % (overlays,
2185 os.path.join(instance.build_dir,
2186 "sanitycheck", "testcase_extra.conf")))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002187
2188 results = self.run_cmake(args)
2189 return results
2190
2191 def build(self):
2192 results = self.run_build(['--build', self.build_dir])
2193 return results
2194
2195 def run(self):
2196
2197 instance = self.instance
2198
2199 if instance.handler.type_str == "device":
2200 instance.handler.suite = self.suite
2201
2202 instance.handler.handle()
2203
Anas Nashif83fc06a2019-06-22 11:04:10 -04002204 sys.stdout.flush()
2205
2206
2207pipeline = queue.LifoQueue()
2208
Anas Nashif11ee5252019-12-04 12:59:10 -05002209
Anas Nashif83fc06a2019-06-22 11:04:10 -04002210class BoundedExecutor(concurrent.futures.ThreadPoolExecutor):
2211 """BoundedExecutor behaves as a ThreadPoolExecutor which will block on
2212 calls to submit() once the limit given as "bound" work items are queued for
2213 execution.
2214 :param bound: Integer - the maximum number of items in the work queue
2215 :param max_workers: Integer - the size of the thread pool
2216 """
Anas Nashifd9882382019-12-12 09:58:28 -05002217
Anas Nashif83fc06a2019-06-22 11:04:10 -04002218 def __init__(self, bound, max_workers, **kwargs):
2219 super().__init__(max_workers)
Anas Nashif11ee5252019-12-04 12:59:10 -05002220 # self.executor = ThreadPoolExecutor(max_workers=max_workers)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002221 self.semaphore = BoundedSemaphore(bound + max_workers)
2222
2223 def submit(self, fn, *args, **kwargs):
2224 self.semaphore.acquire()
2225 try:
2226 future = super().submit(fn, *args, **kwargs)
Anas Nashif11ee5252019-12-04 12:59:10 -05002227 except Exception:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002228 self.semaphore.release()
2229 raise
2230 else:
2231 future.add_done_callback(lambda x: self.semaphore.release())
2232 return future
2233
Andrew Boie6acbe632015-07-17 12:03:52 -07002234
2235class TestSuite:
Andrew Boie60823c22017-02-10 09:43:07 -08002236 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Anas Nashif1c65b6b2018-12-02 19:12:21 -05002237 dt_re = re.compile('([A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Andrew Boie6acbe632015-07-17 12:03:52 -07002238
Anas Nashif83fc06a2019-06-22 11:04:10 -04002239 tc_schema = scl.yaml_load(
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +03002240 os.path.join(ZEPHYR_BASE,
Anas Nashif83fc06a2019-06-22 11:04:10 -04002241 "scripts", "sanity_chk", "testcase-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07002242
Anas Nashif37f9dc52018-02-23 08:53:46 -06002243 def __init__(self, board_root_list, testcase_roots, outdir):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002244
2245 self.roots = testcase_roots
2246 if not isinstance(board_root_list, list):
Anas Nashif11ee5252019-12-04 12:59:10 -05002247 self.board_roots = [board_root_list]
Anas Nashif83fc06a2019-06-22 11:04:10 -04002248 else:
2249 self.board_roots = board_root_list
2250
Anas Nashif56656842019-12-10 12:26:00 -05002251 # Testsuite Options
2252 self.coverage_platform = []
2253 self.build_only = False
2254 self.cmake_only = False
2255 self.enable_slow = False
2256 self.device_testing = False
2257 self.fixture = []
2258 self.enable_coverage = False
2259 self.enable_lsan = False
2260 self.enable_asan = False
2261 self.enable_valgrind = False
2262 self.extra_args = []
Anas Nashife9eb0092019-12-10 16:31:22 -05002263 self.inline_logs = False
Anas Nashifc5ee3952019-12-10 16:38:45 -05002264 self.enable_sizes_report = False
Anas Nashif56656842019-12-10 12:26:00 -05002265
Andrew Boie6acbe632015-07-17 12:03:52 -07002266 # Keep track of which test cases we've filtered out and why
Andrew Boie6acbe632015-07-17 12:03:52 -07002267 self.testcases = {}
2268 self.platforms = []
Anas Nashif5f908822019-11-25 08:19:25 -05002269 self.selected_platforms = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04002270 self.default_platforms = []
Andrew Boieb391e662015-08-31 15:25:45 -07002271 self.outdir = os.path.abspath(outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07002272 self.discards = None
Kumar Galac84235e2018-04-10 13:32:51 -05002273 self.load_errors = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04002274 self.instances = dict()
Andrew Boie6acbe632015-07-17 12:03:52 -07002275
Anas Nashif11ee5252019-12-04 12:59:10 -05002276 self.total_tests = 0 # number of test instances
2277 self.total_cases = 0 # number of test cases
2278 self.total_done = 0 # tests completed
Anas Nashif83fc06a2019-06-22 11:04:10 -04002279 self.total_failed = 0
2280 self.total_skipped = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07002281
Anas Nashif83fc06a2019-06-22 11:04:10 -04002282 self.total_platforms = 0
2283 self.start_time = 0
2284 self.duration = 0
2285 self.warnings = 0
2286 self.cv = threading.Condition()
Anas Nashif61e21632018-04-08 13:30:16 -05002287
Anas Nashif83fc06a2019-06-22 11:04:10 -04002288 # hardcoded for now
2289 self.connected_hardware = []
Andrew Boie3d348712016-04-08 11:52:13 -07002290
Anas Nashif56656842019-12-10 12:26:00 -05002291 def config(self):
2292 logger.info("coverage platform: {}".format(self.coverage_platform))
2293
Anas Nashiff16ed8e2019-12-09 16:22:27 -05002294 # Debug Functions
2295 @staticmethod
2296 def info(what):
2297 sys.stdout.write(what + "\n")
2298 sys.stdout.flush()
2299
Anas Nashif83fc06a2019-06-22 11:04:10 -04002300 def update(self):
2301 self.total_tests = len(self.instances)
2302 self.total_cases = len(self.testcases)
Anas Nashifbd166f42017-09-02 12:32:08 -04002303
Andrew Boie6acbe632015-07-17 12:03:52 -07002304 def compare_metrics(self, filename):
2305 # name, datatype, lower results better
2306 interesting_metrics = [("ram_size", int, True),
2307 ("rom_size", int, True)]
2308
Andrew Boie6acbe632015-07-17 12:03:52 -07002309 if not os.path.exists(filename):
Anas Nashif7a361b82019-12-06 11:37:40 -05002310 logger.info("Cannot compare metrics, %s not found" % filename)
Andrew Boie6acbe632015-07-17 12:03:52 -07002311 return []
2312
2313 results = []
2314 saved_metrics = {}
2315 with open(filename) as fp:
2316 cr = csv.DictReader(fp)
2317 for row in cr:
2318 d = {}
2319 for m, _, _ in interesting_metrics:
2320 d[m] = row[m]
2321 saved_metrics[(row["test"], row["platform"])] = d
2322
Anas Nashif83fc06a2019-06-22 11:04:10 -04002323 for instance in self.instances.values():
2324 mkey = (instance.testcase.name, instance.platform.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07002325 if mkey not in saved_metrics:
2326 continue
2327 sm = saved_metrics[mkey]
2328 for metric, mtype, lower_better in interesting_metrics:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002329 if metric not in instance.metrics:
Andrew Boie6acbe632015-07-17 12:03:52 -07002330 continue
2331 if sm[metric] == "":
2332 continue
Anas Nashif83fc06a2019-06-22 11:04:10 -04002333 delta = instance.metrics.get(metric, 0) - mtype(sm[metric])
Andrew Boieea7928f2015-08-14 14:27:38 -07002334 if delta == 0:
2335 continue
Anas Nashifd9882382019-12-12 09:58:28 -05002336 results.append((instance, metric, instance.metrics.get(metric, 0), delta,
Andrew Boieea7928f2015-08-14 14:27:38 -07002337 lower_better))
Andrew Boie6acbe632015-07-17 12:03:52 -07002338 return results
2339
Anas Nashif83fc06a2019-06-22 11:04:10 -04002340 def misc_reports(self, report, show_footprint, all_deltas,
2341 footprint_threshold, last_metrics):
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002342
Anas Nashif83fc06a2019-06-22 11:04:10 -04002343 if not report:
2344 return
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002345
Anas Nashif83fc06a2019-06-22 11:04:10 -04002346 deltas = self.compare_metrics(report)
2347 warnings = 0
2348 if deltas and show_footprint:
2349 for i, metric, value, delta, lower_better in deltas:
2350 if not all_deltas and ((delta < 0 and lower_better) or
Anas Nashifd9882382019-12-12 09:58:28 -05002351 (delta > 0 and not lower_better)):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002352 continue
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002353
Anas Nashif83fc06a2019-06-22 11:04:10 -04002354 percentage = (float(delta) / float(value - delta))
2355 if not all_deltas and (percentage <
Anas Nashifd9882382019-12-12 09:58:28 -05002356 (footprint_threshold / 100.0)):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002357 continue
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002358
Anas Nashifd9882382019-12-12 09:58:28 -05002359 logger.info("{:<25} {:<60} {}{}{}: {} {:<+4}, is now {:6} {:+.2%}".format(
2360 i.platform.name, i.testcase.name, COLOR_YELLOW,
2361 "INFO" if all_deltas else "WARNING", COLOR_NORMAL,
2362 metric, delta, value, percentage))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002363 warnings += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002364
Anas Nashif83fc06a2019-06-22 11:04:10 -04002365 if warnings:
Anas Nashif7a361b82019-12-06 11:37:40 -05002366 logger.warning("Deltas based on metrics from last %s" %
Anas Nashifd9882382019-12-12 09:58:28 -05002367 ("release" if not last_metrics else "run"))
Anas Nashif61e21632018-04-08 13:30:16 -05002368
Anas Nashif83fc06a2019-06-22 11:04:10 -04002369 def summary(self, unrecognized_sections):
2370 failed = 0
2371 for instance in self.instances.values():
2372 if instance.status == "failed":
2373 failed += 1
2374 elif instance.metrics.get("unrecognized") and not unrecognized_sections:
Anas Nashif7a361b82019-12-06 11:37:40 -05002375 logger.error("%sFAILED%s: %s has unrecognized binary sections: %s" %
Anas Nashifd9882382019-12-12 09:58:28 -05002376 (COLOR_RED, COLOR_NORMAL, instance.name,
2377 str(instance.metrics.get("unrecognized", []))))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002378 failed += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002379
Anas Nashif83fc06a2019-06-22 11:04:10 -04002380 if self.total_tests and self.total_tests != self.total_skipped:
Anas Nashifd9882382019-12-12 09:58:28 -05002381 pass_rate = (float(self.total_tests - self.total_failed - self.total_skipped) / float(
2382 self.total_tests - self.total_skipped))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002383 else:
2384 pass_rate = 0
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002385
Anas Nashifd9882382019-12-12 09:58:28 -05002386 logger.info(
2387 "{}{} of {}{} tests passed ({:.2%}), {}{}{} failed, {} skipped with {}{}{} warnings in {:.2f} seconds".format(
2388 COLOR_RED if failed else COLOR_GREEN,
2389 self.total_tests - self.total_failed - self.total_skipped,
2390 self.total_tests,
2391 COLOR_NORMAL,
2392 pass_rate,
2393 COLOR_RED if self.total_failed else COLOR_NORMAL,
2394 self.total_failed,
2395 COLOR_NORMAL,
2396 self.total_skipped,
2397 COLOR_YELLOW if self.warnings else COLOR_NORMAL,
2398 self.warnings,
2399 COLOR_NORMAL,
2400 self.duration))
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002401
Anas Nashif83fc06a2019-06-22 11:04:10 -04002402 self.total_platforms = len(self.platforms)
2403 if self.platforms:
Anas Nashif7a361b82019-12-06 11:37:40 -05002404 logger.info("In total {} test cases were executed on {} out of total {} platforms ({:02.2f}%)".format(
Anas Nashif83fc06a2019-06-22 11:04:10 -04002405 self.total_cases,
Anas Nashif5f908822019-11-25 08:19:25 -05002406 len(self.selected_platforms),
Anas Nashif83fc06a2019-06-22 11:04:10 -04002407 self.total_platforms,
Anas Nashif5f908822019-11-25 08:19:25 -05002408 (100 * len(self.selected_platforms) / len(self.platforms))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002409 ))
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002410
Anas Nashif56656842019-12-10 12:26:00 -05002411 def save_reports(self, name, report_dir, no_update, release, only_failed):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002412 if not self.instances:
2413 return
Anas Nashif61e21632018-04-08 13:30:16 -05002414
Anas Nashif56656842019-12-10 12:26:00 -05002415 if name:
2416 report_name = name
2417 else:
2418 report_name = "sanitycheck"
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002419
Anas Nashif56656842019-12-10 12:26:00 -05002420 if report_dir:
2421 os.makedirs(report_dir, exist_ok=True)
2422 filename = os.path.join(report_dir, report_name)
2423 outdir = report_dir
Anas Nashif83fc06a2019-06-22 11:04:10 -04002424 else:
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05002425 filename = os.path.join(self.outdir, report_name)
2426 outdir = self.outdir
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002427
Anas Nashif56656842019-12-10 12:26:00 -05002428 if not no_update:
2429 self.xunit_report(filename + ".xml", only_failed)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002430 self.csv_report(filename + ".csv")
2431 self.target_report(outdir)
2432 if self.discards:
2433 self.discard_report(filename + "_discard.csv")
2434
Anas Nashif56656842019-12-10 12:26:00 -05002435 if release:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002436 self.csv_report(RELEASE_DATA)
2437
Anas Nashif83fc06a2019-06-22 11:04:10 -04002438 def add_configurations(self):
2439
2440 for board_root in self.board_roots:
2441 board_root = os.path.abspath(board_root)
2442
Anas Nashif7a361b82019-12-06 11:37:40 -05002443 logger.debug("Reading platform configuration files under %s..." %
Anas Nashifd9882382019-12-12 09:58:28 -05002444 board_root)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002445
2446 for file in glob.glob(os.path.join(board_root, "*", "*", "*.yaml")):
Anas Nashif7a361b82019-12-06 11:37:40 -05002447 logger.debug("Found plaform configuration " + file)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002448 try:
2449 platform = Platform()
2450 platform.load(file)
2451 if platform.sanitycheck:
2452 self.platforms.append(platform)
2453 if platform.default:
2454 self.default_platforms.append(platform.name)
2455
2456 except RuntimeError as e:
Anas Nashif7a361b82019-12-06 11:37:40 -05002457 logger.error("E: %s: can't load: %s" % (file, e))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002458 self.load_errors += 1
2459
Anas Nashifeabaa7f2019-11-18 07:49:17 -08002460 def get_all_tests(self):
2461 tests = []
2462 for _, tc in self.testcases.items():
2463 for case in tc.cases:
2464 tests.append(case)
2465
2466 return tests
2467
Anas Nashif83fc06a2019-06-22 11:04:10 -04002468 @staticmethod
2469 def get_toolchain():
2470 toolchain = os.environ.get("ZEPHYR_TOOLCHAIN_VARIANT", None) or \
2471 os.environ.get("ZEPHYR_GCC_VARIANT", None)
2472
2473 if toolchain == "gccarmemb":
2474 # Remove this translation when gccarmemb is no longer supported.
2475 toolchain = "gnuarmemb"
2476
Anas Nashifb4bdd662018-08-15 17:12:28 -05002477 try:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002478 if not toolchain:
2479 raise SanityRuntimeError("E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined")
Anas Nashifb4bdd662018-08-15 17:12:28 -05002480 except Exception as e:
2481 print(str(e))
2482 sys.exit(2)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002483
Anas Nashif83fc06a2019-06-22 11:04:10 -04002484 return toolchain
2485
Anas Nashif83fc06a2019-06-22 11:04:10 -04002486 def add_testcases(self):
2487 for root in self.roots:
2488 root = os.path.abspath(root)
2489
Anas Nashifd9882382019-12-12 09:58:28 -05002490 logger.debug("Reading test case configuration files under %s..." % root)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002491
2492 for dirpath, dirnames, filenames in os.walk(root, topdown=True):
Anas Nashif7a361b82019-12-06 11:37:40 -05002493 logger.debug("scanning %s" % dirpath)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002494 if 'sample.yaml' in filenames:
2495 filename = 'sample.yaml'
2496 elif 'testcase.yaml' in filenames:
2497 filename = 'testcase.yaml'
2498 else:
2499 continue
2500
Anas Nashif7a361b82019-12-06 11:37:40 -05002501 logger.debug("Found possible test case in " + dirpath)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002502
2503 dirnames[:] = []
2504 tc_path = os.path.join(dirpath, filename)
2505 self.add_testcase(tc_path, root)
2506
2507 def add_testcase(self, tc_data_file, root):
2508 try:
2509 parsed_data = SanityConfigParser(tc_data_file, self.tc_schema)
2510 parsed_data.load()
2511
2512 tc_path = os.path.dirname(tc_data_file)
2513 workdir = os.path.relpath(tc_path, root)
2514
2515 for name in parsed_data.tests.keys():
2516 tc = TestCase()
2517 tc.name = tc.get_unique(root, workdir, name)
2518
2519 tc_dict = parsed_data.get_test(name, testcase_valid_keys)
2520
2521 tc.source_dir = tc_path
2522 tc.yamlfile = tc_data_file
2523
2524 tc.id = name
2525 tc.type = tc_dict["type"]
2526 tc.tags = tc_dict["tags"]
2527 tc.extra_args = tc_dict["extra_args"]
2528 tc.extra_configs = tc_dict["extra_configs"]
2529 tc.arch_whitelist = tc_dict["arch_whitelist"]
2530 tc.arch_exclude = tc_dict["arch_exclude"]
2531 tc.skip = tc_dict["skip"]
2532 tc.platform_exclude = tc_dict["platform_exclude"]
2533 tc.platform_whitelist = tc_dict["platform_whitelist"]
2534 tc.toolchain_exclude = tc_dict["toolchain_exclude"]
2535 tc.toolchain_whitelist = tc_dict["toolchain_whitelist"]
2536 tc.tc_filter = tc_dict["filter"]
2537 tc.timeout = tc_dict["timeout"]
2538 tc.harness = tc_dict["harness"]
2539 tc.harness_config = tc_dict["harness_config"]
2540 tc.build_only = tc_dict["build_only"]
2541 tc.build_on_all = tc_dict["build_on_all"]
2542 tc.slow = tc_dict["slow"]
2543 tc.min_ram = tc_dict["min_ram"]
2544 tc.depends_on = tc_dict["depends_on"]
2545 tc.min_flash = tc_dict["min_flash"]
2546 tc.extra_sections = tc_dict["extra_sections"]
2547
2548 tc.parse_subcases(tc_path)
2549
2550 if tc.name:
2551 self.testcases[tc.name] = tc
2552
2553 except Exception as e:
Anas Nashif7a361b82019-12-06 11:37:40 -05002554 logger.error("%s: can't load (skipping): %s" % (tc_data_file, e))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002555 self.load_errors += 1
2556 return False
2557
2558 return True
2559
Anas Nashif83fc06a2019-06-22 11:04:10 -04002560 def get_platform(self, name):
2561 selected_platform = None
2562 for platform in self.platforms:
2563 if platform.name == name:
2564 selected_platform = platform
2565 break
2566 return selected_platform
2567
2568 def get_last_failed(self):
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05002569 last_run = os.path.join(self.outdir, "sanitycheck.csv")
Anas Nashif83fc06a2019-06-22 11:04:10 -04002570 try:
2571 if not os.path.exists(last_run):
Anas Nashifd9882382019-12-12 09:58:28 -05002572 raise SanityRuntimeError("Couldn't find last sanitycheck run.: %s" % last_run)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002573 except Exception as e:
2574 print(str(e))
2575 sys.exit(2)
2576
2577 total_tests = 0
2578 with open(last_run, "r") as fp:
2579 cr = csv.DictReader(fp)
2580 instance_list = []
2581 for row in cr:
2582 total_tests += 1
2583 if row["passed"] == "True":
2584 continue
2585 test = row["test"]
2586 platform = self.get_platform(row["platform"])
2587 instance = TestInstance(self.testcases[test], platform, self.outdir)
Anas Nashif56656842019-12-10 12:26:00 -05002588 instance.check_build_or_run(
2589 self.build_only,
2590 self.enable_slow,
2591 self.device_testing,
2592 self.fixture
Anas Nashifd9882382019-12-12 09:58:28 -05002593 )
Anas Nashif56656842019-12-10 12:26:00 -05002594 instance.create_overlay(platform, self.enable_asan, self.enable_coverage, self.coverage_platform)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002595 instance_list.append(instance)
2596 self.add_instances(instance_list)
2597
2598 tests_to_run = len(self.instances)
Anas Nashifd9882382019-12-12 09:58:28 -05002599 logger.info("%d tests passed already, retrying %d tests" % (total_tests - tests_to_run, tests_to_run))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002600
2601 def load_from_file(self, file):
2602 try:
2603 if not os.path.exists(file):
2604 raise SanityRuntimeError(
2605 "Couldn't find input file with list of tests.")
2606 except Exception as e:
2607 print(str(e))
2608 sys.exit(2)
2609
2610 with open(file, "r") as fp:
2611 cr = csv.DictReader(fp)
2612 instance_list = []
2613 for row in cr:
2614 if row["arch"] == "arch":
2615 continue
2616 test = row["test"]
2617 platform = self.get_platform(row["platform"])
2618 instance = TestInstance(self.testcases[test], platform, self.outdir)
Anas Nashif56656842019-12-10 12:26:00 -05002619 instance.check_build_or_run(
2620 self.build_only,
2621 self.enable_slow,
2622 self.device_testing,
2623 self.fixture
Anas Nashifd9882382019-12-12 09:58:28 -05002624 )
Anas Nashif56656842019-12-10 12:26:00 -05002625 instance.create_overlay(platform, self.enable_asan, self.enable_coverage, self.coverage_platform)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002626 instance_list.append(instance)
2627 self.add_instances(instance_list)
2628
Anas Nashiff5e5ef02019-12-06 19:20:48 -05002629 def apply_filters(self, **kwargs):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002630
2631 toolchain = self.get_toolchain()
2632
2633 discards = {}
Anas Nashiff5e5ef02019-12-06 19:20:48 -05002634 platform_filter = kwargs.get('platform')
2635 testcase_filter = kwargs.get('run_individual_tests')
2636 arch_filter = kwargs.get('arch')
2637 tag_filter = kwargs.get('tag')
2638 exclude_tag = kwargs.get('exclude_tag')
2639 all_filter = kwargs.get('all')
2640 device_testing_filter = kwargs.get('device_testing')
2641 force_toolchain = kwargs.get('force_toolchain')
Anas Nashif83fc06a2019-06-22 11:04:10 -04002642
Anas Nashif7a361b82019-12-06 11:37:40 -05002643 logger.debug("platform filter: " + str(platform_filter))
2644 logger.debug(" arch_filter: " + str(arch_filter))
2645 logger.debug(" tag_filter: " + str(tag_filter))
2646 logger.debug(" exclude_tag: " + str(exclude_tag))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002647
2648 default_platforms = False
2649
2650 if platform_filter:
2651 platforms = list(filter(lambda p: p.name in platform_filter, self.platforms))
2652 else:
2653 platforms = self.platforms
2654
Anas Nashiff5e5ef02019-12-06 19:20:48 -05002655 if all_filter:
Anas Nashif7a361b82019-12-06 11:37:40 -05002656 logger.info("Selecting all possible platforms per test case")
Anas Nashif83fc06a2019-06-22 11:04:10 -04002657 # When --all used, any --platform arguments ignored
2658 platform_filter = []
2659 elif not platform_filter:
Anas Nashif7a361b82019-12-06 11:37:40 -05002660 logger.info("Selecting default platforms per test case")
Anas Nashif83fc06a2019-06-22 11:04:10 -04002661 default_platforms = True
2662
Anas Nashif7a361b82019-12-06 11:37:40 -05002663 logger.info("Building initial testcase list...")
Anas Nashif83fc06a2019-06-22 11:04:10 -04002664
2665 for tc_name, tc in self.testcases.items():
2666 # list of instances per testcase, aka configurations.
2667 instance_list = []
2668 for plat in platforms:
2669 instance = TestInstance(tc, plat, self.outdir)
Anas Nashif56656842019-12-10 12:26:00 -05002670 instance.check_build_or_run(
2671 self.build_only,
2672 self.enable_slow,
2673 self.device_testing,
2674 self.fixture
Anas Nashifd9882382019-12-12 09:58:28 -05002675 )
Anas Nashif83fc06a2019-06-22 11:04:10 -04002676
2677 if (plat.arch == "unit") != (tc.type == "unit"):
2678 # Discard silently
2679 continue
2680
Anas Nashiff5e5ef02019-12-06 19:20:48 -05002681 if device_testing_filter and instance.build_only:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002682 discards[instance] = "Not runnable on device"
2683 continue
2684
2685 if tc.skip:
2686 discards[instance] = "Skip filter"
2687 continue
2688
2689 if tc.build_on_all and not platform_filter:
2690 platform_filter = []
2691
2692 if tag_filter and not tc.tags.intersection(tag_filter):
2693 discards[instance] = "Command line testcase tag filter"
2694 continue
2695
2696 if exclude_tag and tc.tags.intersection(exclude_tag):
2697 discards[instance] = "Command line testcase exclude filter"
2698 continue
2699
2700 if testcase_filter and tc_name not in testcase_filter:
2701 discards[instance] = "Testcase name filter"
2702 continue
2703
2704 if arch_filter and plat.arch not in arch_filter:
2705 discards[instance] = "Command line testcase arch filter"
2706 continue
2707
2708 if tc.arch_whitelist and plat.arch not in tc.arch_whitelist:
2709 discards[instance] = "Not in test case arch whitelist"
2710 continue
2711
2712 if tc.arch_exclude and plat.arch in tc.arch_exclude:
2713 discards[instance] = "In test case arch exclude"
2714 continue
2715
2716 if tc.platform_exclude and plat.name in tc.platform_exclude:
2717 discards[instance] = "In test case platform exclude"
2718 continue
2719
2720 if tc.toolchain_exclude and toolchain in tc.toolchain_exclude:
2721 discards[instance] = "In test case toolchain exclude"
2722 continue
2723
2724 if platform_filter and plat.name not in platform_filter:
2725 discards[instance] = "Command line platform filter"
2726 continue
2727
2728 if tc.platform_whitelist and plat.name not in tc.platform_whitelist:
2729 discards[instance] = "Not in testcase platform whitelist"
2730 continue
2731
2732 if tc.toolchain_whitelist and toolchain not in tc.toolchain_whitelist:
2733 discards[instance] = "Not in testcase toolchain whitelist"
2734 continue
2735
2736 if not plat.env_satisfied:
2737 discards[instance] = "Environment ({}) not satisfied".format(", ".join(plat.env))
2738 continue
2739
Anas Nashiff5e5ef02019-12-06 19:20:48 -05002740 if not force_toolchain \
Anas Nashifd9882382019-12-12 09:58:28 -05002741 and toolchain and (toolchain not in plat.supported_toolchains) \
2742 and tc.type != 'unit':
Anas Nashif83fc06a2019-06-22 11:04:10 -04002743 discards[instance] = "Not supported by the toolchain"
2744 continue
2745
2746 if plat.ram < tc.min_ram:
2747 discards[instance] = "Not enough RAM"
2748 continue
2749
2750 if tc.depends_on:
2751 dep_intersection = tc.depends_on.intersection(set(plat.supported))
2752 if dep_intersection != set(tc.depends_on):
2753 discards[instance] = "No hardware support"
2754 continue
2755
2756 if plat.flash < tc.min_flash:
2757 discards[instance] = "Not enough FLASH"
2758 continue
2759
2760 if set(plat.ignore_tags) & tc.tags:
2761 discards[instance] = "Excluded tags per platform"
2762 continue
2763
2764 # if nothing stopped us until now, it means this configuration
2765 # needs to be added.
2766 instance_list.append(instance)
2767
2768 # no configurations, so jump to next testcase
2769 if not instance_list:
2770 continue
2771
2772 # if sanitycheck was launched with no platform options at all, we
2773 # take all default platforms
2774 if default_platforms and not tc.build_on_all:
2775 if tc.platform_whitelist:
2776 a = set(self.default_platforms)
2777 b = set(tc.platform_whitelist)
2778 c = a.intersection(b)
2779 if c:
Anas Nashifd9882382019-12-12 09:58:28 -05002780 aa = list(filter(lambda tc: tc.platform.name in c, instance_list))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002781 self.add_instances(aa)
2782 else:
2783 self.add_instances(instance_list[:1])
2784 else:
Anas Nashifd9882382019-12-12 09:58:28 -05002785 instances = list(filter(lambda tc: tc.platform.default, instance_list))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002786 self.add_instances(instances)
2787
2788 for instance in list(filter(lambda tc: not tc.platform.default, instance_list)):
2789 discards[instance] = "Not a default test platform"
2790
2791 else:
2792 self.add_instances(instance_list)
2793
2794 for _, case in self.instances.items():
Anas Nashif56656842019-12-10 12:26:00 -05002795 case.create_overlay(case.platform, self.enable_asan, self.enable_coverage, self.coverage_platform)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002796
2797 self.discards = discards
Anas Nashif5f908822019-11-25 08:19:25 -05002798 self.selected_platforms = set(p.platform.name for p in self.instances.values())
Anas Nashif83fc06a2019-06-22 11:04:10 -04002799
2800 return discards
2801
2802 def add_instances(self, instance_list):
2803 for instance in instance_list:
2804 self.instances[instance.name] = instance
2805
Anas Nashif56656842019-12-10 12:26:00 -05002806 def add_tasks_to_queue(self, test_only=False):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002807 for instance in self.instances.values():
Anas Nashif56656842019-12-10 12:26:00 -05002808 if test_only:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002809 if instance.run:
2810 pipeline.put({"op": "run", "test": instance, "status": "built"})
2811 else:
2812 if instance.status not in ['passed', 'skipped']:
2813 instance.status = None
2814 pipeline.put({"op": "cmake", "test": instance})
2815
2816 return "DONE FEEDING"
2817
Anas Nashifc5ee3952019-12-10 16:38:45 -05002818 def execute(self):
Anas Nashif83fc06a2019-06-22 11:04:10 -04002819 def calc_one_elf_size(instance):
2820 if instance.status not in ["failed", "skipped"]:
2821 if instance.platform.type != "native":
2822 size_calc = instance.calculate_sizes()
2823 instance.metrics["ram_size"] = size_calc.get_ram_size()
2824 instance.metrics["rom_size"] = size_calc.get_rom_size()
2825 instance.metrics["unrecognized"] = size_calc.unrecognized_sections()
2826 else:
2827 instance.metrics["ram_size"] = 0
2828 instance.metrics["rom_size"] = 0
2829 instance.metrics["unrecognized"] = []
2830
2831 instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0
2832
Anas Nashif7a361b82019-12-06 11:37:40 -05002833 logger.info("Adding tasks to the queue...")
Anas Nashif83fc06a2019-06-22 11:04:10 -04002834 # We can use a with statement to ensure threads are cleaned up promptly
2835 with BoundedExecutor(bound=self.jobs, max_workers=self.jobs) as executor:
2836
2837 # start a future for a thread which sends work in through the queue
2838 future_to_test = {
Anas Nashifd9882382019-12-12 09:58:28 -05002839 executor.submit(self.add_tasks_to_queue, self.test_only): 'FEEDER DONE'}
Anas Nashif83fc06a2019-06-22 11:04:10 -04002840
2841 while future_to_test:
2842 # check for status of the futures which are currently working
2843 done, _ = concurrent.futures.wait(
Anas Nashifd9882382019-12-12 09:58:28 -05002844 future_to_test, timeout=0.25,
2845 return_when=concurrent.futures.FIRST_COMPLETED)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002846
2847 # if there is incoming work, start a new future
2848 while not pipeline.empty():
2849 # fetch a url from the queue
2850 message = pipeline.get()
2851 test = message['test']
2852
2853 # Start the load operation and mark the future with its URL
Anas Nashif56656842019-12-10 12:26:00 -05002854 pb = ProjectBuilder(self,
Anas Nashifd9882382019-12-12 09:58:28 -05002855 test,
2856 lsan=self.enable_lsan,
2857 asan=self.enable_asan,
2858 coverage=self.enable_coverage,
2859 extra_args=self.extra_args,
2860 device_testing=self.device_testing,
2861 cmake_only=self.cmake_only,
2862 valgrind=self.enable_valgrind,
2863 inline_logs=self.inline_logs
2864 )
Anas Nashif83fc06a2019-06-22 11:04:10 -04002865 future_to_test[executor.submit(pb.process, message)] = test.name
2866
2867 # process any completed futures
2868 for future in done:
2869 test = future_to_test[future]
2870 try:
2871 data = future.result()
2872 except Exception as exc:
Anas Nashif4f043862019-11-05 06:01:49 -08002873 sys.exit('%r generated an exception: %s' % (test, exc))
2874
Anas Nashif83fc06a2019-06-22 11:04:10 -04002875 else:
2876 if data:
Anas Nashif7a361b82019-12-06 11:37:40 -05002877 logger.debug(data)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002878
2879 # remove the now completed future
2880 del future_to_test[future]
2881
Anas Nashifc5ee3952019-12-10 16:38:45 -05002882 if self.enable_size_report and not self.cmake_only:
Anas Nashif83fc06a2019-06-22 11:04:10 -04002883 # Parallelize size calculation
2884 executor = concurrent.futures.ThreadPoolExecutor(self.jobs)
2885 futures = [executor.submit(calc_one_elf_size, instance)
2886 for instance in self.instances.values()]
2887 concurrent.futures.wait(futures)
2888 else:
2889 for instance in self.instances.values():
2890 instance.metrics["ram_size"] = 0
2891 instance.metrics["rom_size"] = 0
2892 instance.metrics["handler_time"] = instance.handler.duration if instance.handler else 0
2893 instance.metrics["unrecognized"] = []
2894
Anas Nashif83fc06a2019-06-22 11:04:10 -04002895 def discard_report(self, filename):
2896
2897 try:
2898 if self.discards is None:
2899 raise SanityRuntimeError("apply_filters() hasn't been run!")
2900 except Exception as e:
Anas Nashif7a361b82019-12-06 11:37:40 -05002901 logger.error(str(e))
Anas Nashif83fc06a2019-06-22 11:04:10 -04002902 sys.exit(2)
2903
2904 with open(filename, "wt") as csvfile:
2905 fieldnames = ["test", "arch", "platform", "reason"]
2906 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2907 cw.writeheader()
2908 for instance, reason in sorted(self.discards.items()):
2909 rowdict = {"test": instance.testcase.name,
2910 "arch": instance.platform.arch,
2911 "platform": instance.platform.name,
2912 "reason": reason}
2913 cw.writerow(rowdict)
2914
Anas Nashif83fc06a2019-06-22 11:04:10 -04002915 def target_report(self, outdir):
2916 run = "Sanitycheck"
2917 eleTestsuite = None
2918
Anas Nashifd9882382019-12-12 09:58:28 -05002919 platforms = {inst.platform.name for _, inst in self.instances.items()}
Anas Nashif83fc06a2019-06-22 11:04:10 -04002920 for platform in platforms:
2921 errors = 0
2922 passes = 0
2923 fails = 0
2924 duration = 0
2925 skips = 0
2926 for _, instance in self.instances.items():
2927 if instance.platform.name != platform:
2928 continue
2929
2930 handler_time = instance.metrics.get('handler_time', 0)
2931 duration += handler_time
2932 for k in instance.results.keys():
2933 if instance.results[k] == 'PASS':
2934 passes += 1
2935 elif instance.results[k] == 'BLOCK':
2936 errors += 1
2937 elif instance.results[k] == 'SKIP':
2938 skips += 1
2939 else:
2940 fails += 1
2941
2942 eleTestsuites = ET.Element('testsuites')
2943 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
Anas Nashifd9882382019-12-12 09:58:28 -05002944 name=run, time="%f" % duration,
2945 tests="%d" % (errors + passes + fails),
2946 failures="%d" % fails,
2947 errors="%d" % errors, skipped="%d" % skips)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002948
2949 handler_time = 0
2950
2951 # print out test results
2952 for _, instance in self.instances.items():
2953 if instance.platform.name != platform:
2954 continue
2955 handler_time = instance.metrics.get('handler_time', 0)
2956 for k in instance.results.keys():
2957 eleTestcase = ET.SubElement(
Anas Nashifd9882382019-12-12 09:58:28 -05002958 eleTestsuite, 'testcase',
2959 classname="%s:%s" % (instance.platform.name, os.path.basename(instance.testcase.name)),
2960 name="%s" % (k), time="%f" % handler_time)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002961 if instance.results[k] in ['FAIL', 'BLOCK']:
2962 el = None
2963
2964 if instance.results[k] == 'FAIL':
2965 el = ET.SubElement(
2966 eleTestcase,
2967 'failure',
2968 type="failure",
2969 message="failed")
2970 elif instance.results[k] == 'BLOCK':
2971 el = ET.SubElement(
2972 eleTestcase,
2973 'error',
2974 type="failure",
2975 message="failed")
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05002976 p = os.path.join(self.outdir, instance.platform.name, instance.testcase.name)
Anas Nashif83fc06a2019-06-22 11:04:10 -04002977 log_file = os.path.join(p, "handler.log")
2978
2979 if os.path.exists(log_file):
2980 with open(log_file, "rb") as f:
2981 log = f.read().decode("utf-8")
2982 filtered_string = ''.join(filter(lambda x: x in string.printable, log))
2983 el.text = filtered_string
2984
2985 elif instance.results[k] == 'SKIP':
2986 el = ET.SubElement(
2987 eleTestcase,
2988 'skipped',
2989 type="skipped",
2990 message="Skipped")
2991
Anas Nashif83fc06a2019-06-22 11:04:10 -04002992 result = ET.tostring(eleTestsuites)
2993 with open(os.path.join(outdir, platform + ".xml"), 'wb') as f:
2994 f.write(result)
2995
Anas Nashif56656842019-12-10 12:26:00 -05002996 def xunit_report(self, filename, append=False):
Anas Nashifb3311ed2017-04-13 14:44:48 -04002997 fails = 0
2998 passes = 0
2999 errors = 0
Anas Nashif83fc06a2019-06-22 11:04:10 -04003000 skips = 0
3001 duration = 0
Anas Nashifb3311ed2017-04-13 14:44:48 -04003002
Anas Nashif83fc06a2019-06-22 11:04:10 -04003003 for instance in self.instances.values():
3004 handler_time = instance.metrics.get('handler_time', 0)
3005 duration += handler_time
3006 if instance.status == "failed":
3007 if instance.reason in ['build_error', 'handler_crash']:
Anas Nashifb3311ed2017-04-13 14:44:48 -04003008 errors += 1
3009 else:
3010 fails += 1
Anas Nashif83fc06a2019-06-22 11:04:10 -04003011 elif instance.status == 'skipped':
3012 skips += 1
Anas Nashifb3311ed2017-04-13 14:44:48 -04003013 else:
3014 passes += 1
3015
3016 run = "Sanitycheck"
3017 eleTestsuite = None
Anas Nashifb3311ed2017-04-13 14:44:48 -04003018
Anas Nashif83fc06a2019-06-22 11:04:10 -04003019 # When we re-run the tests, we re-use the results and update only with
3020 # the newly run tests.
Anas Nashif0605fa32017-05-07 08:51:02 -04003021 if os.path.exists(filename) and append:
Anas Nashifb3311ed2017-04-13 14:44:48 -04003022 tree = ET.parse(filename)
3023 eleTestsuites = tree.getroot()
Anas Nashif3ba1d432017-12-05 15:28:44 -05003024 eleTestsuite = tree.findall('testsuite')[0]
Anas Nashifb3311ed2017-04-13 14:44:48 -04003025 else:
3026 eleTestsuites = ET.Element('testsuites')
Anas Nashif3ba1d432017-12-05 15:28:44 -05003027 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
Anas Nashif83fc06a2019-06-22 11:04:10 -04003028 name=run, time="%f" % duration,
3029 tests="%d" % (errors + passes + fails + skips),
Anas Nashif3ba1d432017-12-05 15:28:44 -05003030 failures="%d" % fails,
Anas Nashifd9882382019-12-12 09:58:28 -05003031 errors="%d" % (errors), skip="%s" % (skips))
Anas Nashifb3311ed2017-04-13 14:44:48 -04003032
Anas Nashif83fc06a2019-06-22 11:04:10 -04003033 for instance in self.instances.values():
Anas Nashifb3311ed2017-04-13 14:44:48 -04003034
Anas Nashif83fc06a2019-06-22 11:04:10 -04003035 # remove testcases that are a re-run
Anas Nashifb3311ed2017-04-13 14:44:48 -04003036 if append:
3037 for tc in eleTestsuite.findall('testcase'):
Anas Nashif3ba1d432017-12-05 15:28:44 -05003038 if tc.get('classname') == "%s:%s" % (
Anas Nashif83fc06a2019-06-22 11:04:10 -04003039 instance.platform.name, instance.testcase.name):
Anas Nashifb3311ed2017-04-13 14:44:48 -04003040 eleTestsuite.remove(tc)
3041
Anas Nashif83fc06a2019-06-22 11:04:10 -04003042 handler_time = 0
3043 if instance.status != "failed" and instance.handler:
3044 handler_time = instance.metrics.get("handler_time", 0)
Anas Nashifb3311ed2017-04-13 14:44:48 -04003045
Anas Nashifd9882382019-12-12 09:58:28 -05003046
Anas Nashif3ba1d432017-12-05 15:28:44 -05003047 eleTestcase = ET.SubElement(
Anas Nashifd9882382019-12-12 09:58:28 -05003048 eleTestsuite,
3049 'testcase',
3050 classname="%s:%s" % (instance.platform.name, instance.testcase.name),
3051 name="%s" % (instance.testcase.name),
3052 time="%f" % handler_time)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003053
3054 if instance.status == "failed":
Anas Nashif3ba1d432017-12-05 15:28:44 -05003055 failure = ET.SubElement(
3056 eleTestcase,
3057 'failure',
3058 type="failure",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003059 message=instance.reason)
Anas Nashif9e7a2cd2019-12-05 13:07:27 -05003060 p = ("%s/%s/%s" % (self.outdir, instance.platform.name, instance.testcase.name))
Anas Nashifb3311ed2017-04-13 14:44:48 -04003061 bl = os.path.join(p, "build.log")
Anas Nashifc96c90a2019-02-05 07:38:32 -05003062 hl = os.path.join(p, "handler.log")
3063 log_file = bl
Anas Nashif83fc06a2019-06-22 11:04:10 -04003064 if instance.reason != 'Build error':
Anas Nashifc96c90a2019-02-05 07:38:32 -05003065 if os.path.exists(hl):
3066 log_file = hl
3067 else:
3068 log_file = bl
Anas Nashifaccc8eb2017-05-01 16:33:43 -04003069
Anas Nashifc96c90a2019-02-05 07:38:32 -05003070 if os.path.exists(log_file):
3071 with open(log_file, "rb") as f:
Anas Nashif712d3452017-12-29 22:09:03 -05003072 log = f.read().decode("utf-8")
Anas Nashifa4c368e2018-10-15 09:45:59 -04003073 filtered_string = ''.join(filter(lambda x: x in string.printable, log))
3074 failure.text = filtered_string
Anas Nashifba4643b2018-09-23 09:41:59 -05003075 f.close()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003076 elif instance.status == "skipped":
Anas Nashifd9882382019-12-12 09:58:28 -05003077 ET.SubElement(eleTestcase, 'skipped', type="skipped", message="Skipped")
Anas Nashifb3311ed2017-04-13 14:44:48 -04003078
3079 result = ET.tostring(eleTestsuites)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003080 with open(filename, 'wb') as report:
3081 report.write(result)
Anas Nashifb3311ed2017-04-13 14:44:48 -04003082
Anas Nashif83fc06a2019-06-22 11:04:10 -04003083 def csv_report(self, filename):
Andrew Boie08ce5a52016-02-22 13:28:10 -08003084 with open(filename, "wt") as csvfile:
Andrew Boie6acbe632015-07-17 12:03:52 -07003085 fieldnames = ["test", "arch", "platform", "passed", "status",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003086 "extra_args", "handler", "handler_time", "ram_size",
Andrew Boie6acbe632015-07-17 12:03:52 -07003087 "rom_size"]
3088 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
3089 cw.writeheader()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003090 for instance in sorted(self.instances.values()):
3091 rowdict = {"test": instance.testcase.name,
3092 "arch": instance.platform.arch,
3093 "platform": instance.platform.name,
3094 "extra_args": " ".join(instance.testcase.extra_args),
3095 "handler": instance.platform.simulation}
3096
3097 if instance.status in ["failed", "timeout"]:
Andrew Boie6acbe632015-07-17 12:03:52 -07003098 rowdict["passed"] = False
Anas Nashif83fc06a2019-06-22 11:04:10 -04003099 rowdict["status"] = instance.reason
Andrew Boie6acbe632015-07-17 12:03:52 -07003100 else:
3101 rowdict["passed"] = True
Anas Nashif83fc06a2019-06-22 11:04:10 -04003102 if instance.handler:
3103 rowdict["handler_time"] = instance.metrics.get("handler_time", 0)
3104 ram_size = instance.metrics.get("ram_size", 0)
3105 rom_size = instance.metrics.get("rom_size", 0)
3106 rowdict["ram_size"] = ram_size
3107 rowdict["rom_size"] = rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -07003108 cw.writerow(rowdict)
3109
Anas Nashif19ca7832019-11-18 08:16:21 -08003110 def get_testcase(self, identifier):
3111 results = []
3112 for _, tc in self.testcases.items():
3113 for case in tc.cases:
3114 if case == identifier:
3115 results.append(tc)
3116 return results
3117
3118
Andrew Boie6acbe632015-07-17 12:03:52 -07003119def parse_arguments():
Anas Nashif3ba1d432017-12-05 15:28:44 -05003120 parser = argparse.ArgumentParser(
3121 description=__doc__,
3122 formatter_class=argparse.RawDescriptionHelpFormatter)
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -05003123 parser.fromfile_prefix_chars = "+"
Andrew Boie6acbe632015-07-17 12:03:52 -07003124
Marc Herbert932a33a2019-03-12 11:37:53 -07003125 case_select = parser.add_argument_group("Test case selection",
3126 """
3127Artificially long but functional example:
3128 $ ./scripts/sanitycheck -v \\
Marc Herberte5cedca2019-04-08 14:02:34 -07003129 --testcase-root tests/ztest/base \\
3130 --testcase-root tests/kernel \\
Marc Herbert932a33a2019-03-12 11:37:53 -07003131 --test tests/ztest/base/testing.ztest.verbose_0 \\
3132 --test tests/kernel/fifo/fifo_api/kernel.fifo.poll
3133
3134 "kernel.fifo.poll" is one of the test section names in
3135 __/fifo_api/testcase.yaml
3136 """)
Marc Herbertedf17592019-03-08 12:39:11 -08003137
Anas Nashif07d54c02018-07-21 19:29:08 -05003138 parser.add_argument("--force-toolchain", action="store_true",
Anas Nashifd9882382019-12-12 09:58:28 -05003139 help="Do not filter based on toolchain, use the set "
3140 " toolchain unconditionally")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003141 parser.add_argument(
3142 "-p", "--platform", action="append",
3143 help="Platform filter for testing. This option may be used multiple "
Anas Nashifd9882382019-12-12 09:58:28 -05003144 "times. Testcases will only be built/run on the platforms "
3145 "specified. If this option is not used, then platforms marked "
3146 "as default in the platform metadata file will be chosen "
3147 "to build and test. ")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003148 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003149 "-a", "--arch", action="append",
3150 help="Arch filter for testing. Takes precedence over --platform. "
Anas Nashifd9882382019-12-12 09:58:28 -05003151 "If unspecified, test all arches. Multiple invocations "
3152 "are treated as a logical 'or' relationship")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003153 parser.add_argument(
3154 "-t", "--tag", action="append",
3155 help="Specify tags to restrict which tests to run by tag value. "
Anas Nashifd9882382019-12-12 09:58:28 -05003156 "Default is to not do any tag filtering. Multiple invocations "
3157 "are treated as a logical 'or' relationship")
Anas Nashifdfa86e22016-10-24 17:08:56 -04003158 parser.add_argument("-e", "--exclude-tag", action="append",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003159 help="Specify tags of tests that should not run. "
Anas Nashifd9882382019-12-12 09:58:28 -05003160 "Default is to run all tests with all tags.")
Marc Herbertedf17592019-03-08 12:39:11 -08003161 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003162 "-f",
3163 "--only-failed",
3164 action="store_true",
3165 help="Run only those tests that failed the previous sanity check "
Anas Nashifd9882382019-12-12 09:58:28 -05003166 "invocation.")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003167
Anas Nashif3ba1d432017-12-05 15:28:44 -05003168 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003169 "--retry-failed", type=int, default=0,
3170 help="Retry failing tests again, up to the number of times specified.")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003171
Marc Herbert0c465bb2019-03-11 17:28:36 -07003172 test_xor_subtest = case_select.add_mutually_exclusive_group()
3173
3174 test_xor_subtest.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003175 "-s", "--test", action="append",
3176 help="Run only the specified test cases. These are named by "
Anas Nashifd9882382019-12-12 09:58:28 -05003177 "<path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003178
Marc Herbert0c465bb2019-03-11 17:28:36 -07003179 test_xor_subtest.add_argument(
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003180 "--sub-test", action="append",
Marc Herbert932a33a2019-03-12 11:37:53 -07003181 help="""Recursively find sub-test functions and run the entire
3182 test section where they were found, including all sibling test
3183 functions. Sub-tests are named by:
3184 section.name.in.testcase.yaml.function_name_without_test_prefix
3185 Example: kernel.fifo.poll.fifo_loop
3186 """)
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05003187
Anas Nashif3ba1d432017-12-05 15:28:44 -05003188 parser.add_argument(
3189 "-l", "--all", action="store_true",
3190 help="Build/test on all platforms. Any --platform arguments "
Anas Nashifd9882382019-12-12 09:58:28 -05003191 "ignored.")
Andrew Boie6acbe632015-07-17 12:03:52 -07003192
Anas Nashif3ba1d432017-12-05 15:28:44 -05003193 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003194 "-o", "--report-dir",
3195 help="""Output reports containing results of the test run into the
3196 specified directory.
3197 The output will be both in CSV and JUNIT format
3198 (sanitycheck.csv and sanitycheck.xml).
3199 """)
3200
Anas Nashif3ba1d432017-12-05 15:28:44 -05003201 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003202 "--report-name",
3203 help="""Create a report with a custom name.
3204 """)
3205
Anas Nashif83fc06a2019-06-22 11:04:10 -04003206 parser.add_argument("--report-excluded",
Anas Nashifd9882382019-12-12 09:58:28 -05003207 action="store_true",
3208 help="""List all tests that are never run based on current scope and
Anas Nashif83fc06a2019-06-22 11:04:10 -04003209 coverage. If you are looking for accurate results, run this with
3210 --all, but this will take a while...""")
3211
Daniel Leung7f850102016-04-08 11:07:32 -07003212 parser.add_argument("--compare-report",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003213 help="Use this report file for size comparison")
Daniel Leung7f850102016-04-08 11:07:32 -07003214
Anas Nashif3ba1d432017-12-05 15:28:44 -05003215 parser.add_argument(
3216 "-B", "--subset",
3217 help="Only run a subset of the tests, 1/4 for running the first 25%%, "
Anas Nashifd9882382019-12-12 09:58:28 -05003218 "3/5 means run the 3rd fifth of the total. "
3219 "This option is useful when running a large number of tests on "
3220 "different hosts to speed up execution time.")
Anas Nashifa8a13882017-12-30 13:01:06 -05003221
3222 parser.add_argument(
3223 "-N", "--ninja", action="store_true",
Anas Nashif25f6ab62018-03-06 07:15:11 -06003224 help="Use the Ninja generator with CMake")
Anas Nashifa8a13882017-12-30 13:01:06 -05003225
Anas Nashif3ba1d432017-12-05 15:28:44 -05003226 parser.add_argument(
3227 "-y", "--dry-run", action="store_true",
Anas Nashif12d8cce2019-11-20 03:47:27 -08003228 help="""Create the filtered list of test cases, but don't actually
3229 run them. Useful if you're just interested in the discard report
3230 generated for every run and saved in the specified output
3231 directory (sanitycheck_discard.csv).
3232 """)
Andrew Boie6acbe632015-07-17 12:03:52 -07003233
Anas Nashif75547e22018-02-24 08:32:14 -06003234 parser.add_argument("--list-tags", action="store_true",
Anas Nashifd9882382019-12-12 09:58:28 -05003235 help="list all tags in selected tests")
Anas Nashif75547e22018-02-24 08:32:14 -06003236
Marc Herbertedf17592019-03-08 12:39:11 -08003237 case_select.add_argument("--list-tests", action="store_true",
Anas Nashifd9882382019-12-12 09:58:28 -05003238 help="""List of all sub-test functions recursively found in
Marc Herbert932a33a2019-03-12 11:37:53 -07003239 all --testcase-root arguments. Note different sub-tests can share
3240 the same section name and come from different directories.
3241 The output is flattened and reports --sub-test names only,
3242 not their directories. For instance net.socket.getaddrinfo_ok
3243 and net.socket.fd_set belong to different directories.
3244 """)
Anas Nashifc0149cc2018-04-14 23:12:58 -05003245
Anas Nashif434995c2019-12-01 13:55:11 -05003246 case_select.add_argument("--test-tree", action="store_true",
Anas Nashifd9882382019-12-12 09:58:28 -05003247 help="""Output the testsuite in a tree form""")
Anas Nashif434995c2019-12-01 13:55:11 -05003248
Anas Nashif19ca7832019-11-18 08:16:21 -08003249 case_select.add_argument("--list-test-duplicates", action="store_true",
Anas Nashifd9882382019-12-12 09:58:28 -05003250 help="""List tests with duplicate identifiers.
Anas Nashif19ca7832019-11-18 08:16:21 -08003251 """)
3252
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05003253 parser.add_argument("--export-tests", action="store",
Anas Nashifd9882382019-12-12 09:58:28 -05003254 metavar="FILENAME",
3255 help="Export tests case meta-data to a file in CSV format.")
Anas Nashife0a6a0b2018-02-15 20:07:24 -06003256
Anas Nashif654ec5982019-04-11 08:38:21 -04003257 parser.add_argument("--timestamps",
Anas Nashifd9882382019-12-12 09:58:28 -05003258 action="store_true",
3259 help="Print all messages with time stamps")
Anas Nashif654ec5982019-04-11 08:38:21 -04003260
Anas Nashif3ba1d432017-12-05 15:28:44 -05003261 parser.add_argument(
3262 "-r", "--release", action="store_true",
3263 help="Update the benchmark database with the results of this test "
Anas Nashifd9882382019-12-12 09:58:28 -05003264 "run. Intended to be run by CI when tagging an official "
3265 "release. This database is used as a basis for comparison "
3266 "when looking for deltas in metrics such as footprint")
Andrew Boie6acbe632015-07-17 12:03:52 -07003267 parser.add_argument("-w", "--warnings-as-errors", action="store_true",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003268 help="Treat warning conditions as errors")
3269 parser.add_argument(
3270 "-v",
3271 "--verbose",
3272 action="count",
3273 default=0,
3274 help="Emit debugging information, call multiple times to increase "
Anas Nashifd9882382019-12-12 09:58:28 -05003275 "verbosity")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003276 parser.add_argument(
3277 "-i", "--inline-logs", action="store_true",
3278 help="Upon test failure, print relevant log data to stdout "
Anas Nashifd9882382019-12-12 09:58:28 -05003279 "instead of just a path to it")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -08003280 parser.add_argument("--log-file", metavar="FILENAME", action="store",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003281 help="log also to file")
3282 parser.add_argument(
3283 "-m", "--last-metrics", action="store_true",
3284 help="Instead of comparing metrics from the last --release, "
Anas Nashifd9882382019-12-12 09:58:28 -05003285 "compare with the results of the previous sanity check "
3286 "invocation")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003287 parser.add_argument(
3288 "-u",
3289 "--no-update",
3290 action="store_true",
3291 help="do not update the results of the last run of the sanity "
Anas Nashifd9882382019-12-12 09:58:28 -05003292 "checks")
Marc Herbertedf17592019-03-08 12:39:11 -08003293 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003294 "-F",
3295 "--load-tests",
3296 metavar="FILENAME",
3297 action="store",
Marc Herbert932a33a2019-03-12 11:37:53 -07003298 help="Load list of tests and platforms to be run from file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04003299
Marc Herbertedf17592019-03-08 12:39:11 -08003300 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003301 "-E",
3302 "--save-tests",
3303 metavar="FILENAME",
3304 action="store",
Marc Herbert682961a2019-03-20 16:48:49 -07003305 help="Append list of tests and platforms to be run to file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04003306
Andy Doancbecadd2019-02-08 10:19:10 -06003307 test_or_build = parser.add_mutually_exclusive_group()
3308 test_or_build.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003309 "-b", "--build-only", action="store_true",
3310 help="Only build the code, do not execute any of it in QEMU")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003311
Andy Doancbecadd2019-02-08 10:19:10 -06003312 test_or_build.add_argument(
3313 "--test-only", action="store_true",
3314 help="""Only run device tests with current artifacts, do not build
3315 the code""")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003316 parser.add_argument(
Kumar Gala84cf9dc2019-06-15 09:36:06 -05003317 "--cmake-only", action="store_true",
Anas Nashif83fc06a2019-06-22 11:04:10 -04003318 help="Only run cmake, do not build or run.")
Kumar Gala84cf9dc2019-06-15 09:36:06 -05003319
3320 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003321 "-j", "--jobs", type=int,
Marc Herbert9e573382019-07-03 12:49:42 -07003322 help="Number of jobs for building, defaults to number of CPU threads, "
Anas Nashifd9882382019-12-12 09:58:28 -05003323 "overcommited by factor 2 when --build-only")
Anas Nashif73440ea2018-02-19 10:57:03 -06003324
3325 parser.add_argument(
Anas Nashifd9882382019-12-12 09:58:28 -05003326 "--show-footprint", action="store_true",
3327 help="Show footprint statistics and deltas since last release."
3328 )
Anas Nashif424a3db2018-02-20 08:37:24 -06003329 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003330 "-H", "--footprint-threshold", type=float, default=5,
3331 help="When checking test case footprint sizes, warn the user if "
Anas Nashifd9882382019-12-12 09:58:28 -05003332 "the new app size is greater then the specified percentage "
3333 "from the last release. Default is 5. 0 to warn on any "
3334 "increase on app size")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003335 parser.add_argument(
3336 "-D", "--all-deltas", action="store_true",
3337 help="Show all footprint deltas, positive or negative. Implies "
Anas Nashifd9882382019-12-12 09:58:28 -05003338 "--footprint-threshold=0")
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003339 parser.add_argument(
3340 "-O", "--outdir",
Anas Nashifd9882382019-12-12 09:58:28 -05003341 default=os.path.join(os.getcwd(), "sanity-out"),
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00003342 help="Output directory for logs and binaries. "
Anas Nashifd9882382019-12-12 09:58:28 -05003343 "Default is 'sanity-out' in the current directory. "
3344 "This directory will be deleted unless '--no-clean' is set.")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003345 parser.add_argument(
3346 "-n", "--no-clean", action="store_true",
3347 help="Do not delete the outdir before building. Will result in "
Anas Nashifd9882382019-12-12 09:58:28 -05003348 "faster compilation since builds will be incremental")
Marc Herbertedf17592019-03-08 12:39:11 -08003349 case_select.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05003350 "-T", "--testcase-root", action="append", default=[],
3351 help="Base directory to recursively search for test cases. All "
Anas Nashifd9882382019-12-12 09:58:28 -05003352 "testcase.yaml files under here will be processed. May be "
3353 "called multiple times. Defaults to the 'samples/' and "
3354 "'tests/' directories at the base of the Zephyr tree.")
Anas Nashif7dd19ea2018-11-14 08:46:49 -05003355
Anas Nashif3ba1d432017-12-05 15:28:44 -05003356 board_root_list = ["%s/boards" % ZEPHYR_BASE,
3357 "%s/scripts/sanity_chk/boards" % ZEPHYR_BASE]
Anas Nashif7dd19ea2018-11-14 08:46:49 -05003358
Anas Nashif3ba1d432017-12-05 15:28:44 -05003359 parser.add_argument(
Thomas Stilwell53105102019-10-23 14:25:09 +02003360 "-A", "--board-root", action="append", default=board_root_list,
Anas Nashif83fc06a2019-06-22 11:04:10 -04003361 help="""Directory to search for board configuration files. All .yaml
3362files in the directory will be processed. The directory should have the same
3363structure in the main Zephyr tree: boards/<arch>/<board_name>/""")
3364
Anas Nashif3ba1d432017-12-05 15:28:44 -05003365 parser.add_argument(
3366 "-z", "--size", action="append",
3367 help="Don't run sanity checks. Instead, produce a report to "
Anas Nashifd9882382019-12-12 09:58:28 -05003368 "stdout detailing RAM/ROM sizes on the specified filenames. "
3369 "All other command line arguments ignored.")
Anas Nashif3ba1d432017-12-05 15:28:44 -05003370 parser.add_argument(
3371 "-S", "--enable-slow", action="store_true",
3372 help="Execute time-consuming test cases that have been marked "
Anas Nashifd9882382019-12-12 09:58:28 -05003373 "as 'slow' in testcase.yaml. Normally these are only built.")
Sebastian Bøe03ed0952018-11-13 13:36:19 +01003374 parser.add_argument(
3375 "--disable-unrecognized-section-test", action="store_true",
3376 default=False,
3377 help="Skip the 'unrecognized section' test.")
Andrew Boie55121052016-07-20 11:52:04 -07003378 parser.add_argument("-R", "--enable-asserts", action="store_true",
Kumar Gala0d48cbe2018-01-26 10:22:20 -06003379 default=True,
Andrew Boie29599f62018-05-24 13:33:09 -07003380 help="deprecated, left for compatibility")
Kumar Gala0d48cbe2018-01-26 10:22:20 -06003381 parser.add_argument("--disable-asserts", action="store_false",
3382 dest="enable_asserts",
Andrew Boie29599f62018-05-24 13:33:09 -07003383 help="deprecated, left for compatibility")
Anas Nashife3febe92016-11-30 14:25:44 -05003384 parser.add_argument("-Q", "--error-on-deprecations", action="store_false",
Anas Nashif3ba1d432017-12-05 15:28:44 -05003385 help="Error on deprecation warnings.")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003386 parser.add_argument("--enable-size-report", action="store_true",
3387 help="Enable expensive computation of RAM/ROM segment sizes.")
Sebastian Bøec2182612017-11-09 12:25:02 +01003388
3389 parser.add_argument(
3390 "-x", "--extra-args", action="append", default=[],
Alberto Escolar Piedras25e06362018-02-10 17:40:33 +01003391 help="""Extra CMake cache entries to define when building test cases.
3392 May be called multiple times. The key-value entries will be
Sebastian Bøec2182612017-11-09 12:25:02 +01003393 prefixed with -D before being passed to CMake.
3394
3395 E.g
3396 "sanitycheck -x=USE_CCACHE=0"
3397 will translate to
3398 "cmake -DUSE_CCACHE=0"
3399
3400 which will ultimately disable ccache.
3401 """
3402 )
Michael Scott421ce462019-06-18 09:37:46 -07003403
Andy Doan79c48842019-02-08 10:09:04 -06003404 parser.add_argument(
Anas Nashif83fc06a2019-06-22 11:04:10 -04003405 "--device-testing", action="store_true",
3406 help="Test on device directly. Specify the serial device to "
3407 "use with the --device-serial option.")
3408
3409 parser.add_argument(
3410 "-X", "--fixture", action="append", default=[],
3411 help="Specify a fixture that a board might support")
3412 parser.add_argument(
3413 "--device-serial",
3414 help="Serial device for accessing the board (e.g., /dev/ttyACM0)")
3415
3416 parser.add_argument("--generate-hardware-map",
3417 help="""Probe serial devices connected to this platform
3418 and create a hardware map file to be used with
3419 --device-testing
3420 """)
3421
3422 parser.add_argument("--hardware-map",
3423 help="""Load hardware map from a file. This will be used
3424 for testing on hardware that is listed in the file.
3425 """)
3426
3427 parser.add_argument(
Andy Doan79c48842019-02-08 10:09:04 -06003428 "--west-flash", nargs='?', const=[],
3429 help="""Uses west instead of ninja or make to flash when running with
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +02003430 --device-testing. Supports comma-separated argument list.
Andy Doan79c48842019-02-08 10:09:04 -06003431
Michael Scott4ca54392019-07-09 14:21:30 -07003432 E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0
Jorgen Kvalvaagd50fb312019-09-02 15:25:20 +02003433 --west-flash="--board-id=foobar,--erase"
3434 will translate to "west flash -- --board-id=foobar --erase"
Michael Scott4ca54392019-07-09 14:21:30 -07003435
3436 NOTE: device-testing must be enabled to use this option.
Andy Doan79c48842019-02-08 10:09:04 -06003437 """
3438 )
Michael Scott421ce462019-06-18 09:37:46 -07003439 parser.add_argument(
3440 "--west-runner",
3441 help="""Uses the specified west runner instead of default when running
3442 with --west-flash.
3443
3444 E.g "sanitycheck --device-testing --device-serial /dev/ttyACM0
3445 --west-flash --west-runner=pyocd"
3446 will translate to "west flash --runner pyocd"
3447
3448 NOTE: west-flash must be enabled to use this option.
3449 """
3450 )
Jan Van Winkel21212f32019-09-12 00:03:35 +02003451
3452 valgrind_asan_group = parser.add_mutually_exclusive_group()
3453
3454 valgrind_asan_group.add_argument(
Anas Nashifc1ea4522019-10-11 07:32:45 -07003455 "--enable-valgrind", action="store_true",
3456 help="""Run binary through valgrind and check for several memory access
Jan Van Winkel21212f32019-09-12 00:03:35 +02003457 errors. Valgrind needs to be installed on the host. This option only
Anas Nashifc1ea4522019-10-11 07:32:45 -07003458 works with host binaries such as those generated for the native_posix
Jan Van Winkel21212f32019-09-12 00:03:35 +02003459 configuration and is mutual exclusive with --enable-asan.
3460 """)
3461
3462 valgrind_asan_group.add_argument(
3463 "--enable-asan", action="store_true",
3464 help="""Enable address sanitizer to check for several memory access
3465 errors. Libasan needs to be installed on the host. This option only
3466 works with host binaries such as those generated for the native_posix
3467 configuration and is mutual exclusive with --enable-valgrind.
3468 """)
3469
3470 parser.add_argument(
3471 "--enable-lsan", action="store_true",
3472 help="""Enable leak sanitizer to check for heap memory leaks.
3473 Libasan needs to be installed on the host. This option only
3474 works with host binaries such as those generated for the native_posix
3475 configuration and when --enable-asan is given.
Anas Nashifc1ea4522019-10-11 07:32:45 -07003476 """)
Andrew Boie8047a6f2019-07-02 15:43:29 -07003477
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003478 parser.add_argument("--enable-coverage", action="store_true",
Anas Nashif8d72bb92018-11-07 23:05:42 -05003479 help="Enable code coverage using gcov.")
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003480
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003481 parser.add_argument("-C", "--coverage", action="store_true",
Andrew Boie8047a6f2019-07-02 15:43:29 -07003482 help="Generate coverage reports. Implies "
Anas Nashifd9882382019-12-12 09:58:28 -05003483 "--enable_coverage.")
Andrew Boie6acbe632015-07-17 12:03:52 -07003484
Andrew Boie8047a6f2019-07-02 15:43:29 -07003485 parser.add_argument("--coverage-platform", action="append", default=[],
Anas Nashifdbd76492018-11-23 20:24:19 -05003486 help="Plarforms to run coverage reports on. "
Anas Nashifd9882382019-12-12 09:58:28 -05003487 "This option may be used multiple times. "
3488 "Default to what was selected with --platform.")
Anas Nashifdbd76492018-11-23 20:24:19 -05003489
Anas Nashif83fc06a2019-06-22 11:04:10 -04003490 parser.add_argument("--gcov-tool", default=None,
3491 help="Path to the gcov tool to use for code coverage "
Anas Nashifd9882382019-12-12 09:58:28 -05003492 "reports")
Anas Nashif83fc06a2019-06-22 11:04:10 -04003493
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003494 parser.add_argument("--coverage-tool", choices=['lcov', 'gcovr'], default='lcov',
3495 help="Tool to use to generate coverage report.")
3496
Andrew Boie6acbe632015-07-17 12:03:52 -07003497 return parser.parse_args()
3498
Anas Nashifd9882382019-12-12 09:58:28 -05003499
Andrew Boiebbd670c2015-08-17 13:16:11 -07003500def size_report(sc):
Anas Nashif7a361b82019-12-06 11:37:40 -05003501 logger.info(sc.filename)
3502 logger.info("SECTION NAME VMA LMA SIZE HEX SZ TYPE")
Andrew Boie9882dcd2015-10-07 14:25:51 -07003503 for i in range(len(sc.sections)):
3504 v = sc.sections[i]
3505
Anas Nashif7a361b82019-12-06 11:37:40 -05003506 logger.info("%-17s 0x%08x 0x%08x %8d 0x%05x %-7s" %
Anas Nashifd9882382019-12-12 09:58:28 -05003507 (v["name"], v["virt_addr"], v["load_addr"], v["size"], v["size"],
3508 v["type"]))
Andrew Boie9882dcd2015-10-07 14:25:51 -07003509
Anas Nashif7a361b82019-12-06 11:37:40 -05003510 logger.info("Totals: %d bytes (ROM), %d bytes (RAM)" %
Anas Nashifd9882382019-12-12 09:58:28 -05003511 (sc.rom_size, sc.ram_size))
Anas Nashif7a361b82019-12-06 11:37:40 -05003512 logger.info("")
Andrew Boiebbd670c2015-08-17 13:16:11 -07003513
Anas Nashifd9882382019-12-12 09:58:28 -05003514
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003515class CoverageTool:
3516 """ Base class for every supported coverage tool
3517 """
3518
3519 def __init__(self):
3520 self.gcov_tool = options.gcov_tool
3521
3522 @staticmethod
3523 def factory(tool):
3524 if tool == 'lcov':
3525 return Lcov()
3526 if tool == 'gcovr':
3527 return Gcovr()
Anas Nashif7a361b82019-12-06 11:37:40 -05003528 logger.error("Unsupported coverage tool specified: {}".format(tool))
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003529
3530 @staticmethod
3531 def retrieve_gcov_data(intput_file):
3532 if VERBOSE:
Anas Nashifd9882382019-12-12 09:58:28 -05003533 logger.debug("Working on %s" % intput_file)
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003534 extracted_coverage_info = {}
3535 capture_data = False
3536 capture_complete = False
3537 with open(intput_file, 'r') as fp:
3538 for line in fp.readlines():
3539 if re.search("GCOV_COVERAGE_DUMP_START", line):
3540 capture_data = True
3541 continue
3542 if re.search("GCOV_COVERAGE_DUMP_END", line):
3543 capture_complete = True
3544 break
3545 # Loop until the coverage data is found.
3546 if not capture_data:
3547 continue
3548 if line.startswith("*"):
3549 sp = line.split("<")
3550 if len(sp) > 1:
3551 # Remove the leading delimiter "*"
3552 file_name = sp[0][1:]
3553 # Remove the trailing new line char
3554 hex_dump = sp[1][:-1]
3555 else:
3556 continue
Anas Nashifdb9592a2018-10-08 10:19:41 -04003557 else:
3558 continue
Anas Nashifd9882382019-12-12 09:58:28 -05003559 extracted_coverage_info.update({file_name: hex_dump})
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003560 if not capture_data:
3561 capture_complete = True
3562 return {'complete': capture_complete, 'data': extracted_coverage_info}
3563
3564 @staticmethod
3565 def create_gcda_files(extracted_coverage_info):
3566 if VERBOSE:
Anas Nashif6d66a7a2019-12-08 12:11:43 -05003567 logger.debug("Generating gcda files")
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003568 for filename, hexdump_val in extracted_coverage_info.items():
3569 # if kobject_hash is given for coverage gcovr fails
3570 # hence skipping it problem only in gcovr v4.1
3571 if "kobject_hash" in filename:
Anas Nashifd9882382019-12-12 09:58:28 -05003572 filename = (filename[:-4]) + "gcno"
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003573 try:
3574 os.remove(filename)
3575 except Exception:
3576 pass
Anas Nashifdb9592a2018-10-08 10:19:41 -04003577 continue
Anas Nashifdb9592a2018-10-08 10:19:41 -04003578
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003579 with open(filename, 'wb') as fp:
3580 fp.write(bytes.fromhex(hexdump_val))
Anas Nashifdb9592a2018-10-08 10:19:41 -04003581
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003582 def generate(self, outdir):
3583 for filename in glob.glob("%s/**/handler.log" % outdir, recursive=True):
3584 gcov_data = self.__class__.retrieve_gcov_data(filename)
3585 capture_complete = gcov_data['complete']
3586 extracted_coverage_info = gcov_data['data']
3587 if capture_complete:
3588 self.__class__.create_gcda_files(extracted_coverage_info)
Anas Nashif7a361b82019-12-06 11:37:40 -05003589 logger.debug("Gcov data captured: {}".format(filename))
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003590 else:
Anas Nashif7a361b82019-12-06 11:37:40 -05003591 logger.error("Gcov data capture incomplete: {}".format(filename))
Anas Nashifdb9592a2018-10-08 10:19:41 -04003592
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003593 with open(os.path.join(outdir, "coverage.log"), "a") as coveragelog:
3594 ret = self._generate(outdir, coveragelog)
3595 if ret == 0:
Anas Nashif7a361b82019-12-06 11:37:40 -05003596 logger.info("HTML report generated: {}".format(
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003597 os.path.join(outdir, "coverage", "index.html")))
Anas Nashifdb9592a2018-10-08 10:19:41 -04003598
Anas Nashif47cf4bf2019-01-24 21:50:59 -05003599
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003600class Lcov(CoverageTool):
3601
3602 def __init__(self):
3603 super().__init__()
3604 self.ignores = []
3605
3606 def add_ignore_file(self, pattern):
3607 self.ignores.append('*' + pattern + '*')
3608
3609 def add_ignore_directory(self, pattern):
3610 self.ignores.append(pattern + '/*')
3611
3612 def _generate(self, outdir, coveragelog):
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003613 coveragefile = os.path.join(outdir, "coverage.info")
3614 ztestfile = os.path.join(outdir, "ztest.info")
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003615 subprocess.call(["lcov", "--gcov-tool", self.gcov_tool,
3616 "--capture", "--directory", outdir,
3617 "--rc", "lcov_branch_coverage=1",
3618 "--output-file", coveragefile], stdout=coveragelog)
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03003619 # We want to remove tests/* and tests/ztest/test/* but save tests/ztest
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003620 subprocess.call(["lcov", "--gcov-tool", self.gcov_tool, "--extract",
3621 coveragefile,
Anas Nashif3ba1d432017-12-05 15:28:44 -05003622 os.path.join(ZEPHYR_BASE, "tests", "ztest", "*"),
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003623 "--output-file", ztestfile,
3624 "--rc", "lcov_branch_coverage=1"], stdout=coveragelog)
3625
Anas Nashif3cbffef2018-11-07 23:50:54 -05003626 if os.path.exists(ztestfile) and os.path.getsize(ztestfile) > 0:
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003627 subprocess.call(["lcov", "--gcov-tool", self.gcov_tool, "--remove",
3628 ztestfile,
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003629 os.path.join(ZEPHYR_BASE, "tests/ztest/test/*"),
3630 "--output-file", ztestfile,
3631 "--rc", "lcov_branch_coverage=1"],
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003632 stdout=coveragelog)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003633 files = [coveragefile, ztestfile]
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003634 else:
Anas Nashif83fc06a2019-06-22 11:04:10 -04003635 files = [coveragefile]
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003636
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003637 for i in self.ignores:
Anas Nashif3ba1d432017-12-05 15:28:44 -05003638 subprocess.call(
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003639 ["lcov", "--gcov-tool", self.gcov_tool, "--remove",
3640 coveragefile, i, "--output-file",
3641 coveragefile, "--rc", "lcov_branch_coverage=1"],
Anas Nashif3ba1d432017-12-05 15:28:44 -05003642 stdout=coveragelog)
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003643
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003644 # The --ignore-errors source option is added to avoid it exiting due to
3645 # samples/application_development/external_lib/
3646 return subprocess.call(["genhtml", "--legend", "--branch-coverage",
3647 "--ignore-errors", "source",
3648 "-output-directory",
3649 os.path.join(outdir, "coverage")] + files,
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01003650 stdout=coveragelog)
Christian Taedckeed22c5e2019-11-23 16:47:33 +01003651
3652
3653class Gcovr(CoverageTool):
3654
3655 def __init__(self):
3656 super().__init__()
3657 self.ignores = []
3658
3659 def add_ignore_file(self, pattern):
3660 self.ignores.append('.*' + pattern + '.*')
3661
3662 def add_ignore_directory(self, pattern):
3663 self.ignores.append(pattern + '/.*')
3664
3665 @staticmethod
3666 def _interleave_list(prefix, list):
3667 tuple_list = [(prefix, item) for item in list]
3668 return [item for sublist in tuple_list for item in sublist]
3669
3670 def _generate(self, outdir, coveragelog):
3671 coveragefile = os.path.join(outdir, "coverage.json")
3672 ztestfile = os.path.join(outdir, "ztest.json")
3673
3674 excludes = Gcovr._interleave_list("-e", self.ignores)
3675
3676 # We want to remove tests/* and tests/ztest/test/* but save tests/ztest
3677 subprocess.call(["gcovr", "-r", ZEPHYR_BASE, "--gcov-executable",
3678 self.gcov_tool, "-e", "tests/*"] + excludes +
3679 ["--json", "-o", coveragefile, outdir],
3680 stdout=coveragelog)
3681
3682 subprocess.call(["gcovr", "-r", ZEPHYR_BASE, "--gcov-executable",
3683 self.gcov_tool, "-f", "tests/ztest", "-e",
3684 "tests/ztest/test/*", "--json", "-o", ztestfile,
3685 outdir], stdout=coveragelog)
3686
3687 if os.path.exists(ztestfile) and os.path.getsize(ztestfile) > 0:
3688 files = [coveragefile, ztestfile]
3689 else:
3690 files = [coveragefile]
3691
3692 subdir = os.path.join(outdir, "coverage")
3693 os.makedirs(subdir, exist_ok=True)
3694
3695 tracefiles = self._interleave_list("--add-tracefile", files)
3696
3697 return subprocess.call(["gcovr", "-r", ZEPHYR_BASE, "--html",
3698 "--html-details"] + tracefiles +
3699 ["-o", os.path.join(subdir, "index.html")],
3700 stdout=coveragelog)
3701
Anas Nashif3ba1d432017-12-05 15:28:44 -05003702
Anas Nashif83fc06a2019-06-22 11:04:10 -04003703def get_generator():
3704 if options.ninja:
3705 generator_cmd = "ninja"
3706 generator = "Ninja"
3707 else:
3708 generator_cmd = "make"
3709 generator = "Unix Makefiles"
3710 return generator_cmd, generator
3711
3712
3713def export_tests(filename, tests):
3714 with open(filename, "wt") as csvfile:
3715 fieldnames = ['section', 'subsection', 'title', 'reference']
3716 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
3717 for test in tests:
3718 data = test.split(".")
Anas Nashif19ca7832019-11-18 08:16:21 -08003719 if len(data) > 1:
3720 subsec = " ".join(data[1].split("_")).title()
3721 rowdict = {
Anas Nashifd9882382019-12-12 09:58:28 -05003722 "section": data[0].capitalize(),
3723 "subsection": subsec,
3724 "title": test,
3725 "reference": test
3726 }
Anas Nashif19ca7832019-11-18 08:16:21 -08003727 cw.writerow(rowdict)
3728 else:
Anas Nashif7a361b82019-12-06 11:37:40 -05003729 logger.info("{} can't be exported".format(test))
Anas Nashif83fc06a2019-06-22 11:04:10 -04003730
Anas Nashif83fc06a2019-06-22 11:04:10 -04003731
3732def native_and_unit_first(a, b):
3733 if a[0].startswith('unit_testing'):
3734 return -1
3735 if b[0].startswith('unit_testing'):
3736 return 1
3737 if a[0].startswith('native_posix'):
3738 return -1
3739 if b[0].startswith('native_posix'):
3740 return 1
Anas Nashifd9882382019-12-12 09:58:28 -05003741 if a[0].split("/", 1)[0].endswith("_bsim"):
Anas Nashif83fc06a2019-06-22 11:04:10 -04003742 return -1
Anas Nashifd9882382019-12-12 09:58:28 -05003743 if b[0].split("/", 1)[0].endswith("_bsim"):
Anas Nashif83fc06a2019-06-22 11:04:10 -04003744 return 1
3745
3746 return (a > b) - (a < b)
3747
3748
Anas Nashif5f908822019-11-25 08:19:25 -05003749class HardwareMap:
3750 manufacturer = [
3751 'ARM',
3752 'SEGGER',
3753 'MBED',
3754 'STMicroelectronics',
3755 'Atmel Corp.',
3756 'Texas Instruments',
3757 'Silicon Labs',
3758 'NXP Semiconductors'
Anas Nashifd9882382019-12-12 09:58:28 -05003759 ]
Anas Nashif5f908822019-11-25 08:19:25 -05003760
3761 runner_mapping = {
3762 'pyocd': [
3763 'DAPLink CMSIS-DAP',
3764 'MBED CMSIS-DAP'
3765 ],
3766 'jlink': [
3767 'J-Link',
3768 'J-Link OB'
3769 ],
3770 'openocd': [
3771 'STM32 STLink', '^XDS110.*'
3772 ]
3773 }
3774
3775 def __init__(self):
3776 self.detected = []
3777 self.connected_hardware = []
3778
3779 def load_device_from_cmdline(self, serial, platform):
3780 device = {
3781 "serial": serial,
3782 "platform": platform,
3783 "counter": 0,
3784 "available": True,
3785 "connected": True
3786 }
3787 self.connected_hardware.append(device)
3788
3789 def load_hardware_map(self, map_file):
3790 with open(map_file, 'r') as stream:
3791 try:
3792 self.connected_hardware = yaml.safe_load(stream)
3793 except yaml.YAMLError as exc:
3794 print(exc)
3795 for i in self.connected_hardware:
3796 i['counter'] = 0
3797
3798 def scan_hw(self):
3799 from serial.tools import list_ports
3800
3801 serial_devices = list_ports.comports()
Anas Nashif6d66a7a2019-12-08 12:11:43 -05003802 logger.info("Scanning connected hardware...")
Anas Nashif5f908822019-11-25 08:19:25 -05003803 for d in serial_devices:
3804 if d.manufacturer in self.manufacturer:
3805
3806 # TI XDS110 can have multiple serial devices for a single board
3807 # assume endpoint 0 is the serial, skip all others
3808 if d.manufacturer == 'Texas Instruments' and not d.location.endswith('0'):
3809 continue
3810 s_dev = {}
3811 s_dev['platform'] = "unknown"
3812 s_dev['id'] = d.serial_number
3813 s_dev['serial'] = d.device
3814 s_dev['product'] = d.product
3815 s_dev['runner'] = 'unknown'
Anas Nashifd9882382019-12-12 09:58:28 -05003816 for runner, _ in self.runner_mapping.items():
Anas Nashif5f908822019-11-25 08:19:25 -05003817 products = self.runner_mapping.get(runner)
3818 if d.product in products:
3819 s_dev['runner'] = runner
3820 continue
3821 # Try regex matching
3822 for p in products:
3823 if re.match(p, d.product):
3824 s_dev['runner'] = runner
3825
3826 s_dev['available'] = True
3827 s_dev['connected'] = True
3828 self.detected.append(s_dev)
3829 else:
Anas Nashifd9882382019-12-12 09:58:28 -05003830 logger.warning("Unsupported device (%s): %s" % (d.manufacturer, d))
Anas Nashif5f908822019-11-25 08:19:25 -05003831
3832 def write_map(self, hwm_file):
3833 # use existing map
3834 if os.path.exists(hwm_file):
3835 with open(hwm_file, 'r') as yaml_file:
3836 hwm = yaml.load(yaml_file, Loader=yaml.FullLoader)
3837 # disconnect everything
3838 for h in hwm:
3839 h['connected'] = False
3840 h['serial'] = None
3841
3842 for d in self.detected:
3843 for h in hwm:
3844 if d['id'] == h['id'] and d['product'] == h['product']:
Anas Nashif5f908822019-11-25 08:19:25 -05003845 h['connected'] = True
3846 h['serial'] = d['serial']
3847 d['match'] = True
3848
Anas Nashifd9882382019-12-12 09:58:28 -05003849 new = list(filter(lambda n: not n.get('match', False), self.detected))
Anas Nashif5f908822019-11-25 08:19:25 -05003850 hwm = hwm + new
3851
Anas Nashif6d66a7a2019-12-08 12:11:43 -05003852 logger.info("Registered devices:")
3853 print("")
Anas Nashife5006d12019-12-01 11:41:22 -05003854 table = []
3855 header = ["Platform", "ID", "Serial device"]
Anas Nashifd9882382019-12-12 09:58:28 -05003856 for p in sorted(hwm, key=lambda i: i['platform']):
Anas Nashife5006d12019-12-01 11:41:22 -05003857 platform = p.get('platform')
3858 table.append([platform, p.get('id', None), p.get('serial')])
3859 print(tabulate(table, headers=header, tablefmt="github"))
3860
Anas Nashif5f908822019-11-25 08:19:25 -05003861 with open(hwm_file, 'w') as yaml_file:
3862 yaml.dump(hwm, yaml_file, default_flow_style=False)
3863
3864 else:
3865 # create new file
3866 with open(hwm_file, 'w') as yaml_file:
3867 yaml.dump(self.detected, yaml_file, default_flow_style=False)
3868
Anas Nashifd9882382019-12-12 09:58:28 -05003869
Anas Nashiff5e5ef02019-12-06 19:20:48 -05003870options = None
Anas Nashif5f908822019-11-25 08:19:25 -05003871
Anas Nashifd9882382019-12-12 09:58:28 -05003872
Andrew Boie6acbe632015-07-17 12:03:52 -07003873def main():
Andrew Boie4b182472015-07-31 12:25:22 -07003874 start_time = time.time()
Anas Nashif7a361b82019-12-06 11:37:40 -05003875 global VERBOSE
Anas Nashife10b6512017-12-30 13:01:45 -05003876 global options
Andrew Boie1578ef72019-07-03 10:19:29 -07003877
Anas Nashiff5e5ef02019-12-06 19:20:48 -05003878 options = parse_arguments()
Anas Nashif7a361b82019-12-06 11:37:40 -05003879
3880 # Cleanup
3881 if options.no_clean or options.only_failed or options.test_only:
3882 if os.path.exists(options.outdir):
3883 logger.info("Keeping artifacts untouched")
3884 elif os.path.exists(options.outdir):
Anas Nashifd9882382019-12-12 09:58:28 -05003885 for i in range(1, 100):
Anas Nashif7a361b82019-12-06 11:37:40 -05003886 new_out = options.outdir + ".{}".format(i)
3887 if not os.path.exists(new_out):
3888 logger.info("Renaming output directory to {}".format(new_out))
3889 shutil.move(options.outdir, new_out)
3890 break
3891
3892 os.makedirs(options.outdir, exist_ok=True)
3893
3894 # create file handler which logs even debug messages
3895 if options.log_file:
3896 fh = logging.FileHandler(options.log_file)
3897 else:
3898 fh = logging.FileHandler(os.path.join(options.outdir, "sanitycheck.log"))
3899
3900 fh.setLevel(logging.DEBUG)
3901
3902 # create console handler with a higher log level
3903 ch = logging.StreamHandler()
3904
Anas Nashif7a361b82019-12-06 11:37:40 -05003905 VERBOSE += options.verbose
3906 if VERBOSE > 1:
3907 ch.setLevel(logging.DEBUG)
3908 else:
3909 ch.setLevel(logging.INFO)
3910
Anas Nashif7a361b82019-12-06 11:37:40 -05003911 # create formatter and add it to the handlers
Anas Nashifd4cef072019-12-08 11:58:00 -05003912 if options.timestamps:
3913 formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
3914 else:
Anas Nashif6d66a7a2019-12-08 12:11:43 -05003915 formatter = logging.Formatter('%(levelname)-7s - %(message)s')
Anas Nashifd4cef072019-12-08 11:58:00 -05003916
Anas Nashif7a361b82019-12-06 11:37:40 -05003917 formatter_file = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
3918 ch.setFormatter(formatter)
3919 fh.setFormatter(formatter_file)
3920
3921 # add the handlers to logger
3922 logger.addHandler(ch)
3923 logger.addHandler(fh)
Andrew Boiebbd670c2015-08-17 13:16:11 -07003924
Anas Nashif5f908822019-11-25 08:19:25 -05003925 hwm = HardwareMap()
Anas Nashif83fc06a2019-06-22 11:04:10 -04003926 if options.generate_hardware_map:
Anas Nashif5f908822019-11-25 08:19:25 -05003927 hwm.scan_hw()
3928 hwm.write_map(options.generate_hardware_map)
Anas Nashif83fc06a2019-06-22 11:04:10 -04003929 return
3930
Anas Nashif5f908822019-11-25 08:19:25 -05003931 if not options.device_testing and options.hardware_map:
3932 hwm.load_hardware_map(options.hardware_map)
3933
Anas Nashif6d66a7a2019-12-08 12:11:43 -05003934 logger.info("Available devices:")
3935 print("")
Anas Nashif5f908822019-11-25 08:19:25 -05003936 table = []
3937 header = ["Platform", "ID", "Serial device"]
3938 for p in hwm.connected_hardware:
3939 platform = p.get('platform')
3940 if p['connected']:
3941 table.append([platform, p.get('id', None), p['serial']])
3942 print(tabulate(table, headers=header, tablefmt="github"))
3943 return
Anas Nashif83fc06a2019-06-22 11:04:10 -04003944
Michael Scott421ce462019-06-18 09:37:46 -07003945 if options.west_runner and not options.west_flash:
Anas Nashif7a361b82019-12-06 11:37:40 -05003946 logger.error("west-runner requires west-flash to be enabled")
Michael Scott421ce462019-06-18 09:37:46 -07003947 sys.exit(1)
3948
Michael Scott4ca54392019-07-09 14:21:30 -07003949 if options.west_flash and not options.device_testing:
Anas Nashif7a361b82019-12-06 11:37:40 -05003950 logger.error("west-flash requires device-testing to be enabled")
Michael Scott4ca54392019-07-09 14:21:30 -07003951 sys.exit(1)
3952
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003953 if options.coverage:
3954 options.enable_coverage = True
Christian Taedckec415a4e2019-11-23 23:25:36 +01003955
3956 if not options.coverage_platform:
3957 options.coverage_platform = options.platform
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02003958
Anas Nashife10b6512017-12-30 13:01:45 -05003959 if options.size:
3960 for fn in options.size:
Andrew Boie52fef672016-11-29 12:21:59 -08003961 size_report(SizeCalculator(fn, []))
Andrew Boiebbd670c2015-08-17 13:16:11 -07003962 sys.exit(0)
3963
Anas Nashife10b6512017-12-30 13:01:45 -05003964 if options.subset:
3965 subset, sets = options.subset.split("/")
Anas Nashif035799f2017-05-13 21:31:53 -04003966 if int(subset) > 0 and int(sets) >= int(subset):
Anas Nashif7a361b82019-12-06 11:37:40 -05003967 logger.info("Running only a subset: %s/%s" % (subset, sets))
Anas Nashif035799f2017-05-13 21:31:53 -04003968 else:
Anas Nashif7a361b82019-12-06 11:37:40 -05003969 logger.error("You have provided a wrong subset value: %s." % options.subset)
Anas Nashif035799f2017-05-13 21:31:53 -04003970 return
3971
Anas Nashife10b6512017-12-30 13:01:45 -05003972 if not options.testcase_root:
3973 options.testcase_root = [os.path.join(ZEPHYR_BASE, "tests"),
Anas Nashifd9882382019-12-12 09:58:28 -05003974 os.path.join(ZEPHYR_BASE, "samples")]
Andrew Boie3d348712016-04-08 11:52:13 -07003975
Anas Nashife0d931f2019-12-09 15:23:43 -05003976 if options.show_footprint or options.compare_report or options.release:
3977 options.enable_size_report = True
3978
Anas Nashif56656842019-12-10 12:26:00 -05003979 suite = TestSuite(options.board_root, options.testcase_root, options.outdir)
3980
3981 # Set testsuite options from command line.
3982 suite.build_only = options.build_only
Anas Nashifc5ee3952019-12-10 16:38:45 -05003983 suite.cmake_only = options.cmake_only
3984 suite.test_only = options.test_only
Anas Nashif56656842019-12-10 12:26:00 -05003985 suite.enable_slow = options.enable_slow
3986 suite.device_testing = options.device_testing
3987 suite.fixture = options.fixture
3988 suite.enable_asan = options.enable_asan
3989 suite.enable_lsan = options.enable_lsan
3990 suite.enable_coverage = options.enable_coverage
Anas Nashif56656842019-12-10 12:26:00 -05003991 suite.enable_valgrind = options.enable_valgrind
3992 suite.coverage_platform = options.coverage_platform
Anas Nashife9eb0092019-12-10 16:31:22 -05003993 suite.inline_logs = options.inline_logs
Anas Nashifc5ee3952019-12-10 16:38:45 -05003994 suite.enable_size_report = options.enable_size_report
Anas Nashif51ae4ed2019-12-05 11:02:02 -05003995
3996 # Set number of jobs
3997 if options.jobs:
3998 suite.jobs = options.jobs
3999 elif options.build_only:
4000 suite.jobs = multiprocessing.cpu_count() * 2
4001 else:
4002 suite.jobs = multiprocessing.cpu_count()
Anas Nashif7a361b82019-12-06 11:37:40 -05004003 logger.info("JOBS: %d" % suite.jobs)
Anas Nashif51ae4ed2019-12-05 11:02:02 -05004004
Anas Nashif83fc06a2019-06-22 11:04:10 -04004005 suite.add_testcases()
4006 suite.add_configurations()
Anas Nashifbd166f42017-09-02 12:32:08 -04004007
Anas Nashif83fc06a2019-06-22 11:04:10 -04004008 if options.device_testing:
4009 if options.hardware_map:
Anas Nashif5f908822019-11-25 08:19:25 -05004010 hwm.load_hardware_map(options.hardware_map)
4011 suite.connected_hardware = hwm.connected_hardware
Anas Nashif83fc06a2019-06-22 11:04:10 -04004012 if not options.platform:
4013 options.platform = []
Anas Nashif5f908822019-11-25 08:19:25 -05004014 for platform in hwm.connected_hardware:
Anas Nashif83fc06a2019-06-22 11:04:10 -04004015 if platform['connected']:
4016 options.platform.append(platform['platform'])
4017
Anas Nashifd9882382019-12-12 09:58:28 -05004018 elif options.device_serial: # back-ward compatibility
Anas Nashif83fc06a2019-06-22 11:04:10 -04004019 if options.platform and len(options.platform) == 1:
Anas Nashif5f908822019-11-25 08:19:25 -05004020 hwm.load_device_from_cmdline(options.device_serial, options.platform[0])
Anas Nashifebf8dae2019-12-16 09:22:21 -05004021 suite.connected_hardware = hwm.connected_hardware
Anas Nashif83fc06a2019-06-22 11:04:10 -04004022 else:
Anas Nashif7a361b82019-12-06 11:37:40 -05004023 logger.error("""When --device-testing is used with --device-serial, only one
Anas Nashif83fc06a2019-06-22 11:04:10 -04004024 platform is allowed""")
4025
Anas Nashif83fc06a2019-06-22 11:04:10 -04004026 if suite.load_errors:
Kumar Galac84235e2018-04-10 13:32:51 -05004027 sys.exit(1)
4028
Anas Nashif75547e22018-02-24 08:32:14 -06004029 if options.list_tags:
4030 tags = set()
Anas Nashif83fc06a2019-06-22 11:04:10 -04004031 for _, tc in suite.testcases.items():
Anas Nashif75547e22018-02-24 08:32:14 -06004032 tags = tags.union(tc.tags)
4033
4034 for t in tags:
4035 print("- {}".format(t))
4036
4037 return
4038
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05004039 if options.export_tests:
4040 cnt = 0
Anas Nashifeabaa7f2019-11-18 07:49:17 -08004041 tests = suite.get_all_tests()
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05004042 export_tests(options.export_tests, tests)
4043 return
4044
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004045 run_individual_tests = []
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05004046
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004047 if options.test:
4048 run_individual_tests = options.test
4049
Anas Nashif434995c2019-12-01 13:55:11 -05004050 if options.list_tests or options.test_tree or options.list_test_duplicates or options.sub_test:
Anas Nashif49b22d42019-06-14 13:45:34 -04004051 cnt = 0
Anas Nashifeabaa7f2019-11-18 07:49:17 -08004052 all_tests = suite.get_all_tests()
Anas Nashif49b22d42019-06-14 13:45:34 -04004053
Anas Nashif19ca7832019-11-18 08:16:21 -08004054 if options.list_test_duplicates:
4055 import collections
4056 dupes = [item for item, count in collections.Counter(all_tests).items() if count > 1]
4057 if dupes:
4058 print("Tests with duplicate identifiers:")
4059 for dupe in dupes:
4060 print("- {}".format(dupe))
4061 for dc in suite.get_testcase(dupe):
4062 print(" - {}".format(dc))
4063 else:
4064 print("No duplicates found.")
4065 return
4066
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004067 if options.sub_test:
Anas Nashifd11fd782019-11-18 10:22:56 -08004068 for st in options.sub_test:
4069 subtests = suite.get_testcase(st)
4070 for sti in subtests:
4071 run_individual_tests.append(sti.name)
4072
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004073 if run_individual_tests:
Anas Nashif7a361b82019-12-06 11:37:40 -05004074 logger.info("Running the following tests:")
Anas Nashifeabaa7f2019-11-18 07:49:17 -08004075 for test in run_individual_tests:
4076 print(" - {}".format(test))
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004077 else:
Anas Nashif7a361b82019-12-06 11:37:40 -05004078 logger.info("Tests not found")
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004079 return
4080
Anas Nashif434995c2019-12-01 13:55:11 -05004081 elif options.list_tests or options.test_tree:
4082 if options.test_tree:
4083 testsuite = Node("Testsuite")
4084 samples = Node("Samples", parent=testsuite)
4085 tests = Node("Tests", parent=testsuite)
4086
Anas Nashifc1c3cc62019-12-01 13:24:33 -05004087 for test in sorted(all_tests):
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004088 cnt = cnt + 1
Anas Nashif434995c2019-12-01 13:55:11 -05004089 if options.list_tests:
4090 print(" - {}".format(test))
4091
4092 if options.test_tree:
4093 if test.startswith("sample."):
4094 sec = test.split(".")
4095 area = find(samples, lambda node: node.name == sec[1] and node.parent == samples)
4096 if not area:
4097 area = Node(sec[1], parent=samples)
4098
4099 t = Node(test, parent=area)
4100 else:
4101 sec = test.split(".")
4102 area = find(tests, lambda node: node.name == sec[0] and node.parent == tests)
4103 if not area:
4104 area = Node(sec[0], parent=tests)
4105
4106 if area and len(sec) > 2:
4107 subarea = find(area, lambda node: node.name == sec[1] and node.parent == area)
4108 if not subarea:
4109 subarea = Node(sec[1], parent=area)
4110
4111 t = Node(test, parent=subarea)
4112
4113 if options.list_tests:
4114 print("{} total.".format(cnt))
4115
4116 if options.test_tree:
4117 for pre, _, node in RenderTree(testsuite):
4118 print("%s%s" % (pre, node.name))
4119
Anas Nashif1ea5d7b2018-07-12 09:25:22 -05004120 return
Anas Nashifc0149cc2018-04-14 23:12:58 -05004121
Anas Nashifbd166f42017-09-02 12:32:08 -04004122 discards = []
Anas Nashif83fc06a2019-06-22 11:04:10 -04004123
4124 if options.only_failed:
4125 suite.get_last_failed()
Anas Nashif5f908822019-11-25 08:19:25 -05004126 suite.selected_platforms = set(p.platform.name for p in suite.instances.values())
Anas Nashif83fc06a2019-06-22 11:04:10 -04004127 elif options.load_tests:
4128 suite.load_from_file(options.load_tests)
4129 elif options.test_only:
4130 last_run = os.path.join(options.outdir, "sanitycheck.csv")
4131 suite.load_from_file(last_run)
Anas Nashifbd166f42017-09-02 12:32:08 -04004132 else:
Anas Nashiff5e5ef02019-12-06 19:20:48 -05004133 discards = suite.apply_filters(
4134 build_only=options.build_only,
4135 enable_slow=options.enable_slow,
4136 platform=options.platform,
4137 arch=options.arch,
4138 tag=options.tag,
4139 exclude_tag=options.exclude_tag,
4140 force_toolchain=options.force_toolchain,
4141 all=options.all,
4142 run_individual_tests=run_individual_tests,
4143 device_testing=options.device_testing
4144
4145 )
Andrew Boie6acbe632015-07-17 12:03:52 -07004146
Anas Nashif30551f42018-01-12 21:56:59 -05004147 if VERBOSE > 1 and discards:
Anas Nashiff18e2ab2018-01-13 07:57:42 -05004148 # if we are using command line platform filter, no need to list every
4149 # other platform as excluded, we know that already.
4150 # Show only the discards that apply to the selected platforms on the
4151 # command line
4152
Andrew Boie08ce5a52016-02-22 13:28:10 -08004153 for i, reason in discards.items():
Anas Nashiff18e2ab2018-01-13 07:57:42 -05004154 if options.platform and i.platform.name not in options.platform:
4155 continue
Anas Nashif7a361b82019-12-06 11:37:40 -05004156 logger.debug(
Anas Nashif3ba1d432017-12-05 15:28:44 -05004157 "{:<25} {:<50} {}SKIPPED{}: {}".format(
4158 i.platform.name,
Anas Nashif83fc06a2019-06-22 11:04:10 -04004159 i.testcase.name,
Anas Nashif3ba1d432017-12-05 15:28:44 -05004160 COLOR_YELLOW,
4161 COLOR_NORMAL,
4162 reason))
Andrew Boie6acbe632015-07-17 12:03:52 -07004163
Anas Nashif49b22d42019-06-14 13:45:34 -04004164 if options.report_excluded:
Anas Nashifeabaa7f2019-11-18 07:49:17 -08004165 all_tests = suite.get_all_tests()
Anas Nashif49b22d42019-06-14 13:45:34 -04004166 to_be_run = set()
Anas Nashifd9882382019-12-12 09:58:28 -05004167 for i, p in suite.instances.items():
Anas Nashif83fc06a2019-06-22 11:04:10 -04004168 to_be_run.update(p.testcase.cases)
Anas Nashif49b22d42019-06-14 13:45:34 -04004169
Anas Nashif83fc06a2019-06-22 11:04:10 -04004170 if all_tests - to_be_run:
Anas Nashif49b22d42019-06-14 13:45:34 -04004171 print("Tests that never build or run:")
Anas Nashif83fc06a2019-06-22 11:04:10 -04004172 for not_run in all_tests - to_be_run:
Anas Nashif49b22d42019-06-14 13:45:34 -04004173 print("- {}".format(not_run))
4174
4175 return
4176
Anas Nashife10b6512017-12-30 13:01:45 -05004177 if options.subset:
Anas Nashifd9882382019-12-12 09:58:28 -05004178 # suite.instances = OrderedDict(sorted(suite.instances.items(),
Anas Nashif83fc06a2019-06-22 11:04:10 -04004179 # key=cmp_to_key(native_and_unit_first)))
Anas Nashife10b6512017-12-30 13:01:45 -05004180 subset, sets = options.subset.split("/")
Anas Nashif83fc06a2019-06-22 11:04:10 -04004181 total = len(suite.instances)
Anas Nashif035799f2017-05-13 21:31:53 -04004182 per_set = round(total / int(sets))
Anas Nashif3ba1d432017-12-05 15:28:44 -05004183 start = (int(subset) - 1) * per_set
Anas Nashif035799f2017-05-13 21:31:53 -04004184 if subset == sets:
4185 end = total
4186 else:
4187 end = start + per_set
4188
Anas Nashif83fc06a2019-06-22 11:04:10 -04004189 sliced_instances = islice(suite.instances.items(), start, end)
4190 suite.instances = OrderedDict(sliced_instances)
Anas Nashif035799f2017-05-13 21:31:53 -04004191
Anas Nashif83fc06a2019-06-22 11:04:10 -04004192 if options.save_tests:
4193 suite.csv_report(options.save_tests)
Andrew Boie6acbe632015-07-17 12:03:52 -07004194 return
4195
Anas Nashif7a361b82019-12-06 11:37:40 -05004196 logger.info("%d test configurations selected, %d configurations discarded due to filters." %
Anas Nashifd9882382019-12-12 09:58:28 -05004197 (len(suite.instances), len(discards)))
Anas Nashif83fc06a2019-06-22 11:04:10 -04004198
Peter Bigot3d46ea52019-11-21 12:00:18 -06004199 if options.device_testing:
4200 print("\nDevice testing on:")
Anas Nashif5f908822019-11-25 08:19:25 -05004201 table = []
4202 header = ["Platform", "ID", "Serial device"]
Peter Bigot3d46ea52019-11-21 12:00:18 -06004203 for p in suite.connected_hardware:
Anas Nashif5f908822019-11-25 08:19:25 -05004204 platform = p.get('platform')
4205 if p['connected'] and platform in suite.selected_platforms:
4206 table.append([platform, p.get('id', None), p['serial']])
4207 print(tabulate(table, headers=header, tablefmt="github"))
4208 print("")
Peter Bigot3d46ea52019-11-21 12:00:18 -06004209
Anas Nashif83fc06a2019-06-22 11:04:10 -04004210 if options.dry_run:
4211 duration = time.time() - start_time
Anas Nashif7a361b82019-12-06 11:37:40 -05004212 logger.info("Completed in %d seconds" % (duration))
Anas Nashif83fc06a2019-06-22 11:04:10 -04004213 return
4214
4215 retries = options.retry_failed + 1
4216 completed = 0
4217
4218 suite.update()
4219 suite.start_time = start_time
4220
4221 while True:
4222 completed += 1
4223
4224 if completed > 1:
Anas Nashifd9882382019-12-12 09:58:28 -05004225 logger.info("%d Iteration:" % (completed))
4226 time.sleep(60) # waiting for the system to settle down
Anas Nashif83fc06a2019-06-22 11:04:10 -04004227 suite.total_done = suite.total_tests - suite.total_failed
4228 suite.total_failed = 0
4229
Anas Nashifc5ee3952019-12-10 16:38:45 -05004230 suite.execute()
Anas Nashif7a361b82019-12-06 11:37:40 -05004231 print("")
Andrew Boie6acbe632015-07-17 12:03:52 -07004232
Anas Nashif83fc06a2019-06-22 11:04:10 -04004233 retries = retries - 1
4234 if retries == 0 or suite.total_failed == 0:
4235 break
Anas Nashife0a6a0b2018-02-15 20:07:24 -06004236
Anas Nashif83fc06a2019-06-22 11:04:10 -04004237 suite.misc_reports(options.compare_report, options.show_footprint,
Anas Nashifd9882382019-12-12 09:58:28 -05004238 options.all_deltas, options.footprint_threshold, options.last_metrics)
Andrew Boie6acbe632015-07-17 12:03:52 -07004239
Anas Nashif83a98e52019-11-24 07:42:06 -05004240 suite.duration = time.time() - start_time
4241 suite.summary(options.disable_unrecognized_section_test)
4242
Anas Nashife10b6512017-12-30 13:01:45 -05004243 if options.coverage:
Anas Nashif83fc06a2019-06-22 11:04:10 -04004244 if options.gcov_tool is None:
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02004245 use_system_gcov = False
Andrew Boie49cf4862019-07-08 12:02:13 -07004246
4247 for plat in options.coverage_platform:
Anas Nashif83fc06a2019-06-22 11:04:10 -04004248 ts_plat = suite.get_platform(plat)
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02004249 if ts_plat and (ts_plat.type in {"native", "unit"}):
4250 use_system_gcov = True
Andrew Boie49cf4862019-07-08 12:02:13 -07004251
Alberto Escolar Piedrasaf30f2b2019-09-12 14:44:08 +02004252 if use_system_gcov or "ZEPHYR_SDK_INSTALL_DIR" not in os.environ:
Andrew Boie49cf4862019-07-08 12:02:13 -07004253 options.gcov_tool = "gcov"
4254 else:
4255 options.gcov_tool = os.path.join(os.environ["ZEPHYR_SDK_INSTALL_DIR"],
Anas Nashifd9882382019-12-12 09:58:28 -05004256 "i586-zephyr-elf/bin/i586-zephyr-elf-gcov")
Andrew Boie49cf4862019-07-08 12:02:13 -07004257
Anas Nashif7a361b82019-12-06 11:37:40 -05004258 logger.info("Generating coverage files...")
Christian Taedckeed22c5e2019-11-23 16:47:33 +01004259 coverage_tool = CoverageTool.factory(options.coverage_tool)
4260 coverage_tool.add_ignore_file('generated')
4261 coverage_tool.add_ignore_directory('tests')
4262 coverage_tool.add_ignore_directory('samples')
4263 coverage_tool.generate(options.outdir)
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03004264
Anas Nashif83fc06a2019-06-22 11:04:10 -04004265 if options.device_testing:
4266 print("\nHardware distribution summary:\n")
Anas Nashif5f908822019-11-25 08:19:25 -05004267 table = []
4268 header = ['Board', 'ID', 'Counter']
4269 for p in hwm.connected_hardware:
4270 if p['connected'] and p['platform'] in suite.selected_platforms:
4271 row = [p['platform'], p.get('id', None), p['counter']]
4272 table.append(row)
4273 print(tabulate(table, headers=header, tablefmt="github"))
4274
Anas Nashif56656842019-12-10 12:26:00 -05004275 suite.save_reports(options.report_name,
4276 options.report_dir,
4277 options.no_update,
4278 options.release,
4279 options.only_failed)
4280
Anas Nashif83fc06a2019-06-22 11:04:10 -04004281 if suite.total_failed or (suite.warnings and options.warnings_as_errors):
Andrew Boie6acbe632015-07-17 12:03:52 -07004282 sys.exit(1)
4283
Anas Nashifd9882382019-12-12 09:58:28 -05004284
Andrew Boie6acbe632015-07-17 12:03:52 -07004285if __name__ == "__main__":
4286 main()