blob: 522c5c9b65f5ad7313c88c22d7fc312bcbd640e0 [file] [log] [blame]
Andrew Boie08ce5a52016-02-22 13:28:10 -08001#!/usr/bin/env python3
Anas Nashifa792a3d2017-04-04 18:47:49 -04002# vim: set syntax=python ts=4 :
Andrew Boie6acbe632015-07-17 12:03:52 -07003"""Zephyr Sanity Tests
4
5This script scans for the set of unit test applications in the git
6repository and attempts to execute them. By default, it tries to
7build each test case on one platform per architecture, using a precedence
Paul Sokolovskyff70add2017-06-16 01:31:54 +03008list defined in an architecture configuration file, and if possible
Andrew Boie6acbe632015-07-17 12:03:52 -07009run the tests in the QEMU emulator.
10
Anas Nashifa792a3d2017-04-04 18:47:49 -040011Test cases are detected by the presence of a 'testcase.yaml' or a sample.yaml
12files in the application's project directory. This file may contain one or more
13blocks, each identifying a test scenario. The title of the block is a name for
14the test case, which only needs to be unique for the test cases specified in
15that testcase meta-data. The full canonical name for each test case is <path to
16test case>/<block>.
Andrew Boie6acbe632015-07-17 12:03:52 -070017
Anas Nashif3ba1d432017-12-05 15:28:44 -050018Each test block in the testcase meta data can define the following key/value
19pairs:
Andrew Boie6acbe632015-07-17 12:03:52 -070020
Anas Nashiffa695d22017-10-04 16:14:27 -040021 tags: <list of tags> (required)
Andrew Boie6acbe632015-07-17 12:03:52 -070022 A set of string tags for the testcase. Usually pertains to
23 functional domains but can be anything. Command line invocations
24 of this script can filter the set of tests to run based on tag.
25
Anas Nashifa792a3d2017-04-04 18:47:49 -040026 skip: <True|False> (default False)
Anas Nashif2bd99bc2015-10-12 13:10:57 -040027 skip testcase unconditionally. This can be used for broken tests.
28
Anas Nashifa792a3d2017-04-04 18:47:49 -040029 slow: <True|False> (default False)
Andrew Boie6bb087c2016-02-10 13:39:00 -080030 Don't run this test case unless --enable-slow was passed in on the
31 command line. Intended for time-consuming test cases that are only
32 run under certain circumstances, like daily builds. These test cases
33 are still compiled.
34
Anas Nashifa792a3d2017-04-04 18:47:49 -040035 extra_args: <list of extra arguments>
Sebastian Bøec2182612017-11-09 12:25:02 +010036 Extra cache entries to pass to CMake when building or running the
Andrew Boie6acbe632015-07-17 12:03:52 -070037 test case.
38
Anas Nashifebc329d2017-10-17 09:00:33 -040039 extra_configs: <list of extra configurations>
40 Extra configuration options to be merged with a master prj.conf
41 when building or running the test case.
42
Anas Nashifa792a3d2017-04-04 18:47:49 -040043 build_only: <True|False> (default False)
Andrew Boie6acbe632015-07-17 12:03:52 -070044 If true, don't try to run the test under QEMU even if the
45 selected platform supports it.
46
Anas Nashifa792a3d2017-04-04 18:47:49 -040047 build_on_all: <True|False> (default False)
48 If true, attempt to build test on all available platforms.
49
50 depends_on: <list of features>
51 A board or platform can announce what features it supports, this option
52 will enable the test only those platforms that provide this feature.
53
54 min_ram: <integer>
55 minimum amount of RAM needed for this test to build and run. This is
56 compared with information provided by the board metadata.
57
58 min_flash: <integer>
59 minimum amount of ROM needed for this test to build and run. This is
60 compared with information provided by the board metadata.
61
62 timeout: <number of seconds>
Andrew Boie6acbe632015-07-17 12:03:52 -070063 Length of time to run test in QEMU before automatically killing it.
64 Default to 60 seconds.
65
Anas Nashifa792a3d2017-04-04 18:47:49 -040066 arch_whitelist: <list of arches, such as x86, arm, arc>
Andrew Boie6acbe632015-07-17 12:03:52 -070067 Set of architectures that this test case should only be run for.
68
Anas Nashifa792a3d2017-04-04 18:47:49 -040069 arch_exclude: <list of arches, such as x86, arm, arc>
Anas Nashif30d13872015-10-05 10:02:45 -040070 Set of architectures that this test case should not run on.
71
Anas Nashifa792a3d2017-04-04 18:47:49 -040072 platform_whitelist: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040073 Set of platforms that this test case should only be run for.
74
Anas Nashifa792a3d2017-04-04 18:47:49 -040075 platform_exclude: <list of platforms>
Anas Nashif30d13872015-10-05 10:02:45 -040076 Set of platforms that this test case should not run on.
Andrew Boie6acbe632015-07-17 12:03:52 -070077
Anas Nashifa792a3d2017-04-04 18:47:49 -040078 extra_sections: <list of extra binary sections>
Andrew Boie52fef672016-11-29 12:21:59 -080079 When computing sizes, sanitycheck will report errors if it finds
80 extra, unexpected sections in the Zephyr binary unless they are named
81 here. They will not be included in the size calculation.
82
Anas Nashifa792a3d2017-04-04 18:47:49 -040083 filter: <expression>
Andrew Boie3ea78922016-03-24 14:46:00 -070084 Filter whether the testcase should be run by evaluating an expression
85 against an environment containing the following values:
86
87 { ARCH : <architecture>,
88 PLATFORM : <platform>,
Javier B Perez79414542016-08-08 12:24:59 -050089 <all CONFIG_* key/value pairs in the test's generated defconfig>,
90 *<env>: any environment variable available
Andrew Boie3ea78922016-03-24 14:46:00 -070091 }
92
93 The grammar for the expression language is as follows:
94
95 expression ::= expression "and" expression
96 | expression "or" expression
97 | "not" expression
98 | "(" expression ")"
99 | symbol "==" constant
100 | symbol "!=" constant
101 | symbol "<" number
102 | symbol ">" number
103 | symbol ">=" number
104 | symbol "<=" number
105 | symbol "in" list
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700106 | symbol ":" string
Andrew Boie3ea78922016-03-24 14:46:00 -0700107 | symbol
108
109 list ::= "[" list_contents "]"
110
111 list_contents ::= constant
112 | list_contents "," constant
113
114 constant ::= number
115 | string
116
117
118 For the case where expression ::= symbol, it evaluates to true
119 if the symbol is defined to a non-empty string.
120
121 Operator precedence, starting from lowest to highest:
122
123 or (left associative)
124 and (left associative)
125 not (right associative)
126 all comparison operators (non-associative)
127
128 arch_whitelist, arch_exclude, platform_whitelist, platform_exclude
129 are all syntactic sugar for these expressions. For instance
130
131 arch_exclude = x86 arc
132
133 Is the same as:
134
135 filter = not ARCH in ["x86", "arc"]
Andrew Boie6acbe632015-07-17 12:03:52 -0700136
Andrew Boie7a87f9f2016-06-02 12:27:54 -0700137 The ':' operator compiles the string argument as a regular expression,
138 and then returns a true value only if the symbol's value in the environment
139 matches. For example, if CONFIG_SOC="quark_se" then
140
141 filter = CONFIG_SOC : "quark.*"
142
143 Would match it.
144
Anas Nashifa792a3d2017-04-04 18:47:49 -0400145The set of test cases that actually run depends on directives in the testcase
146filed and options passed in on the command line. If there is any confusion,
147running with -v or --discard-report can help show why particular test cases
148were skipped.
Andrew Boie6acbe632015-07-17 12:03:52 -0700149
150Metrics (such as pass/fail state and binary size) for the last code
151release are stored in scripts/sanity_chk/sanity_last_release.csv.
152To update this, pass the --all --release options.
153
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -0500154To load arguments from a file, write '+' before the file name, e.g.,
155+file_name. File content must be one or more valid arguments separated by
156line break instead of white spaces.
157
Andrew Boie6acbe632015-07-17 12:03:52 -0700158Most everyday users will run with no arguments.
159"""
160
Anas Nashifaae71d72018-04-21 22:26:48 -0500161import contextlib
162import mmap
Andrew Boie6acbe632015-07-17 12:03:52 -0700163import argparse
164import os
165import sys
Andrew Boie6acbe632015-07-17 12:03:52 -0700166import re
Andrew Boie6acbe632015-07-17 12:03:52 -0700167import subprocess
168import multiprocessing
169import select
170import shutil
171import signal
172import threading
173import time
174import csv
Andrew Boie5d4eb782015-10-02 10:04:56 -0700175import glob
Anas Nashif73440ea2018-02-19 10:57:03 -0600176import serial
Daniel Leung6b170072016-04-07 12:10:25 -0700177import concurrent
178import concurrent.futures
Anas Nashifb3311ed2017-04-13 14:44:48 -0400179import xml.etree.ElementTree as ET
Anas Nashife6fcc012017-05-17 09:29:09 -0400180from xml.sax.saxutils import escape
Anas Nashif035799f2017-05-13 21:31:53 -0400181from collections import OrderedDict
182from itertools import islice
Anas Nashif1a5bba72018-01-05 08:07:45 -0500183from functools import cmp_to_key
Anas Nashife24350c2018-07-11 15:09:22 -0500184from pathlib import Path
185from distutils.spawn import find_executable
Andrew Boie6acbe632015-07-17 12:03:52 -0700186
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700187import logging
Anas Nashif3ba1d432017-12-05 15:28:44 -0500188from sanity_chk import scl
189from sanity_chk import expr_parser
190
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700191log_format = "%(levelname)s %(name)s::%(module)s.%(funcName)s():%(lineno)d: %(message)s"
Anas Nashif3ba1d432017-12-05 15:28:44 -0500192logging.basicConfig(format=log_format, level=30)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700193
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +0300194ZEPHYR_BASE = os.environ.get("ZEPHYR_BASE")
195if not ZEPHYR_BASE:
Anas Nashif427cdd32015-08-06 07:25:42 -0400196 sys.stderr.write("$ZEPHYR_BASE environment variable undefined.\n")
Andrew Boie6acbe632015-07-17 12:03:52 -0700197 exit(1)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -0700198
Andrew Boie3ea78922016-03-24 14:46:00 -0700199sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/"))
200
Andrew Boie3ea78922016-03-24 14:46:00 -0700201
Andrew Boie6acbe632015-07-17 12:03:52 -0700202VERBOSE = 0
203LAST_SANITY = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk",
204 "last_sanity.csv")
Anas Nashifb3311ed2017-04-13 14:44:48 -0400205LAST_SANITY_XUNIT = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk",
Anas Nashif3ba1d432017-12-05 15:28:44 -0500206 "last_sanity.xml")
Andrew Boie6acbe632015-07-17 12:03:52 -0700207RELEASE_DATA = os.path.join(ZEPHYR_BASE, "scripts", "sanity_chk",
208 "sanity_last_release.csv")
Oleg Zhurakivskyyf3bc9672018-08-17 18:31:38 +0300209JOBS = multiprocessing.cpu_count() * 2
Andrew Boie6acbe632015-07-17 12:03:52 -0700210
211if os.isatty(sys.stdout.fileno()):
212 TERMINAL = True
213 COLOR_NORMAL = '\033[0m'
214 COLOR_RED = '\033[91m'
215 COLOR_GREEN = '\033[92m'
216 COLOR_YELLOW = '\033[93m'
217else:
218 TERMINAL = False
219 COLOR_NORMAL = ""
220 COLOR_RED = ""
221 COLOR_GREEN = ""
222 COLOR_YELLOW = ""
223
224class SanityCheckException(Exception):
225 pass
226
Anas Nashif3ba1d432017-12-05 15:28:44 -0500227
Andrew Boie6acbe632015-07-17 12:03:52 -0700228class SanityRuntimeError(SanityCheckException):
229 pass
230
Anas Nashif3ba1d432017-12-05 15:28:44 -0500231
Andrew Boie6acbe632015-07-17 12:03:52 -0700232class ConfigurationError(SanityCheckException):
233 def __init__(self, cfile, message):
234 self.cfile = cfile
235 self.message = message
236
237 def __str__(self):
238 return repr(self.cfile + ": " + self.message)
239
Anas Nashif3ba1d432017-12-05 15:28:44 -0500240
Andrew Boie6acbe632015-07-17 12:03:52 -0700241class MakeError(SanityCheckException):
242 pass
243
Anas Nashif3ba1d432017-12-05 15:28:44 -0500244
Andrew Boie6acbe632015-07-17 12:03:52 -0700245class BuildError(MakeError):
246 pass
247
Anas Nashif3ba1d432017-12-05 15:28:44 -0500248
Andrew Boie6acbe632015-07-17 12:03:52 -0700249class ExecutionError(MakeError):
250 pass
251
Anas Nashif3ba1d432017-12-05 15:28:44 -0500252
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800253log_file = None
254
Anas Nashif3ba1d432017-12-05 15:28:44 -0500255
Andrew Boie6acbe632015-07-17 12:03:52 -0700256# Debug Functions
Andrew Boie08ce5a52016-02-22 13:28:10 -0800257def info(what):
258 sys.stdout.write(what + "\n")
Paul Sokolovsky3ba08762017-10-27 14:53:24 +0300259 sys.stdout.flush()
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800260 if log_file:
261 log_file.write(what + "\n")
262 log_file.flush()
Andrew Boie6acbe632015-07-17 12:03:52 -0700263
Anas Nashif3ba1d432017-12-05 15:28:44 -0500264
Andrew Boie6acbe632015-07-17 12:03:52 -0700265def error(what):
266 sys.stderr.write(COLOR_RED + what + COLOR_NORMAL + "\n")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -0800267 if log_file:
268 log_file(what + "\n")
269 log_file.flush()
Andrew Boie6acbe632015-07-17 12:03:52 -0700270
Anas Nashif3ba1d432017-12-05 15:28:44 -0500271
Andrew Boie08ce5a52016-02-22 13:28:10 -0800272def debug(what):
273 if VERBOSE >= 1:
274 info(what)
275
Anas Nashif3ba1d432017-12-05 15:28:44 -0500276
Andrew Boie6acbe632015-07-17 12:03:52 -0700277def verbose(what):
278 if VERBOSE >= 2:
Andrew Boie08ce5a52016-02-22 13:28:10 -0800279 info(what)
Andrew Boie6acbe632015-07-17 12:03:52 -0700280
Anas Nashif576be982017-12-23 20:20:27 -0500281class HarnessImporter:
282
283 def __init__(self, name):
284 sys.path.insert(0, os.path.join(ZEPHYR_BASE, "scripts/sanity_chk"))
285 module = __import__("harness")
286 if name:
287 my_class = getattr(module, name)
288 else:
289 my_class = getattr(module, "Test")
290
291 self.instance = my_class()
Anas Nashif3ba1d432017-12-05 15:28:44 -0500292
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300293class Handler:
Anas Nashif576be982017-12-23 20:20:27 -0500294 def __init__(self, instance):
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300295 """Constructor
296
297 @param name Arbitrary name of the created thread
Anas Nashif66bdb322017-11-25 17:20:07 -0500298 @param outdir Working directory, should be where handler pid file (qemu.pid for example)
299 gets created by the build system
300 @param log_fn Absolute path to write out handler's log data
301 @param timeout Kill the handler process if it doesn't finish up within
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300302 the given number of seconds
303 """
304 self.lock = threading.Lock()
305 self.state = "waiting"
Anas Nashif13773752018-07-06 18:20:23 -0500306 self.run = False
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300307 self.metrics = {}
Anas Nashifc8390f12017-11-25 17:14:12 -0500308 self.metrics["handler_time"] = 0
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300309 self.metrics["ram_size"] = 0
310 self.metrics["rom_size"] = 0
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300311
Anas Nashifdf7ee612018-07-07 06:09:01 -0500312 self.binary = None
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500313 self.call_make_run = False
Anas Nashifdf7ee612018-07-07 06:09:01 -0500314
Anas Nashifd3384fb2018-02-22 06:44:16 -0600315 self.name = instance.name
316 self.instance = instance
317 self.timeout = instance.test.timeout
318 self.sourcedir = instance.test.code_location
319 self.outdir = instance.outdir
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500320 self.log = os.path.join(self.outdir, "handler.log")
Anas Nashifd3384fb2018-02-22 06:44:16 -0600321 self.returncode = 0
322 self.set_state("running", {})
323
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300324 def set_state(self, state, metrics):
325 self.lock.acquire()
326 self.state = state
327 self.metrics.update(metrics)
328 self.lock.release()
329
330 def get_state(self):
331 self.lock.acquire()
332 ret = (self.state, self.metrics)
333 self.lock.release()
334 return ret
335
Anas Nashifdf7ee612018-07-07 06:09:01 -0500336class BinaryHandler(Handler):
337 def __init__(self, instance):
338 """Constructor
339
340 @param instance Test Instance
341 """
342 super().__init__(instance)
343
344 self.valgrind = False
345
346 def _output_reader(self, proc, harness):
347 log_out_fp = open(self.log, "wt")
348 for line in iter(proc.stdout.readline, b''):
349 verbose("OUTPUT: {0}".format(line.decode('utf-8').rstrip()))
350 log_out_fp.write(line.decode('utf-8'))
351 log_out_fp.flush()
352 harness.handle(line.decode('utf-8').rstrip())
353 if harness.state:
354 proc.terminate()
355 break
356
357 log_out_fp.close()
358
359 def handle(self):
360 out_state = "failed"
361
362 harness_name = self.instance.test.harness.capitalize()
363 harness_import = HarnessImporter(harness_name)
364 harness = harness_import.instance
365 harness.configure(self.instance)
366
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500367 if self.call_make_run:
368 if options.ninja:
369 generator_cmd = "ninja"
370 else:
371 generator_cmd = "make"
372 command = [generator_cmd, "-C", self.outdir, "run"]
373 else:
374 command = [self.binary]
Anas Nashifdf7ee612018-07-07 06:09:01 -0500375
376 if shutil.which("valgrind") and self.valgrind:
377 command = ["valgrind", "--error-exitcode=2",
378 "--leak-check=full"] + command
379
380 with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as proc:
381 t = threading.Thread(target=self._output_reader, args=(proc, harness, ))
382 t.start()
383 t.join(self.timeout)
384 if t.is_alive():
385 proc.terminate()
386 out_state = "timeout"
387 t.join()
388
389 proc.wait()
390 self.returncode = proc.returncode
391 if proc.returncode != 0:
392 out_state = "failed"
393
394 if options.enable_coverage:
395 returncode = subprocess.call(["GCOV_PREFIX=" + self.outdir,
396 "gcov", self.sourcedir, "-b", "-s", self.outdir], shell=True)
397
Anas Nashif99f5a6c2018-07-07 08:45:53 -0500398 # FIME: This is needed when killing the simulator, the console is
399 # garbled and needs to be reset. Did not find a better way to do that.
400
401 subprocess.call(["stty", "sane"])
Anas Nashifdf7ee612018-07-07 06:09:01 -0500402 self.instance.results = harness.tests
403 if harness.state:
404 self.set_state(harness.state, {})
405 else:
406 self.set_state(out_state, {})
Anas Nashif73440ea2018-02-19 10:57:03 -0600407
408class DeviceHandler(Handler):
409
410 def __init__(self, instance):
411 """Constructor
412
413 @param instance Test Instance
414 """
415 super().__init__(instance)
416
Anas Nashif73440ea2018-02-19 10:57:03 -0600417 def monitor_serial(self, ser, harness):
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500418 log_out_fp = open(self.log, "wt")
Anas Nashif73440ea2018-02-19 10:57:03 -0600419
420 while ser.isOpen():
Anas Nashif61e21632018-04-08 13:30:16 -0500421 try:
422 serial_line = ser.readline()
423 except TypeError:
424 pass
425
Anas Nashif73440ea2018-02-19 10:57:03 -0600426 if serial_line:
Anas Nashifd3384fb2018-02-22 06:44:16 -0600427 sl = serial_line.decode('utf-8', 'ignore')
428 verbose("DEVICE: {0}".format(sl.rstrip()))
429
430 log_out_fp.write(sl)
431 log_out_fp.flush()
432 harness.handle(sl.rstrip())
Anas Nashif73440ea2018-02-19 10:57:03 -0600433 if harness.state:
434 ser.close()
435 break
436
437 log_out_fp.close()
438
439 def handle(self):
440 out_state = "failed"
441
Anas Nashifd3384fb2018-02-22 06:44:16 -0600442 if options.ninja:
443 generator_cmd = "ninja"
444 else:
445 generator_cmd = "make"
446
447 command = [generator_cmd, "-C", self.outdir, "flash"]
448
Anas Nashif73440ea2018-02-19 10:57:03 -0600449 device = options.device_serial
450 ser = serial.Serial(
451 device,
452 baudrate=115200,
453 parity=serial.PARITY_NONE,
454 stopbits=serial.STOPBITS_ONE,
455 bytesize=serial.EIGHTBITS,
456 timeout=self.timeout
457 )
458
459 ser.flush()
460
461 harness_name = self.instance.test.harness.capitalize()
462 harness_import = HarnessImporter(harness_name)
463 harness = harness_import.instance
464 harness.configure(self.instance)
465
466 t = threading.Thread(target=self.monitor_serial, args=(ser, harness))
467 t.start()
468
Anas Nashif61e21632018-04-08 13:30:16 -0500469 try:
470 subprocess.check_output(command, stderr=subprocess.PIPE)
471 except subprocess.CalledProcessError:
472 pass
Anas Nashif73440ea2018-02-19 10:57:03 -0600473
474 t.join(self.timeout)
475 if t.is_alive():
476 out_state = "timeout"
477 ser.close()
478
479 if ser.isOpen():
480 ser.close()
481
Anas Nashifd3384fb2018-02-22 06:44:16 -0600482 if out_state == "timeout":
483 for c in self.instance.test.cases:
484 if c not in harness.tests:
485 harness.tests[c] = "BLOCK"
Anas Nashif61e21632018-04-08 13:30:16 -0500486
487 self.instance.results = harness.tests
Anas Nashife0a6a0b2018-02-15 20:07:24 -0600488
Anas Nashif73440ea2018-02-19 10:57:03 -0600489 if harness.state:
490 self.set_state(harness.state, {})
491 else:
492 self.set_state(out_state, {})
493
Anas Nashif3ba1d432017-12-05 15:28:44 -0500494
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300495class QEMUHandler(Handler):
Andrew Boie6acbe632015-07-17 12:03:52 -0700496 """Spawns a thread to monitor QEMU output from pipes
497
Anas Nashif3ac7b3a2017-08-03 10:03:02 -0400498 We pass QEMU_PIPE to 'make run' and monitor the pipes for output.
Andrew Boie6acbe632015-07-17 12:03:52 -0700499 We need to do this as once qemu starts, it runs forever until killed.
500 Test cases emit special messages to the console as they run, we check
501 for these to collect whether the test passed or failed.
502 """
Andrew Boie6acbe632015-07-17 12:03:52 -0700503
504 @staticmethod
Anas Nashif576be982017-12-23 20:20:27 -0500505 def _thread(handler, timeout, outdir, logfile, fifo_fn, pid_fn, results, harness):
Andrew Boie6acbe632015-07-17 12:03:52 -0700506 fifo_in = fifo_fn + ".in"
507 fifo_out = fifo_fn + ".out"
508
509 # These in/out nodes are named from QEMU's perspective, not ours
510 if os.path.exists(fifo_in):
511 os.unlink(fifo_in)
512 os.mkfifo(fifo_in)
513 if os.path.exists(fifo_out):
514 os.unlink(fifo_out)
515 os.mkfifo(fifo_out)
516
517 # We don't do anything with out_fp but we need to open it for
518 # writing so that QEMU doesn't block, due to the way pipes work
519 out_fp = open(fifo_in, "wb")
520 # Disable internal buffering, we don't
521 # want read() or poll() to ever block if there is data in there
522 in_fp = open(fifo_out, "rb", buffering=0)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800523 log_out_fp = open(logfile, "wt")
Andrew Boie6acbe632015-07-17 12:03:52 -0700524
525 start_time = time.time()
526 timeout_time = start_time + timeout
527 p = select.poll()
528 p.register(in_fp, select.POLLIN)
Anas Nashif39ae72b2018-08-29 01:45:38 -0400529 out_state = None
Andrew Boie6acbe632015-07-17 12:03:52 -0700530
531 metrics = {}
532 line = ""
533 while True:
534 this_timeout = int((timeout_time - time.time()) * 1000)
535 if this_timeout < 0 or not p.poll(this_timeout):
Anas Nashif39ae72b2018-08-29 01:45:38 -0400536 if not out_state:
537 out_state = "timeout"
Andrew Boie6acbe632015-07-17 12:03:52 -0700538 break
539
Genaro Saucedo Tejada2968b362016-08-09 15:11:53 -0500540 try:
541 c = in_fp.read(1).decode("utf-8")
542 except UnicodeDecodeError:
543 # Test is writing something weird, fail
544 out_state = "unexpected byte"
545 break
546
Andrew Boie6acbe632015-07-17 12:03:52 -0700547 if c == "":
548 # EOF, this shouldn't happen unless QEMU crashes
549 out_state = "unexpected eof"
550 break
551 line = line + c
552 if c != "\n":
553 continue
554
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300555 # line contains a full line of data output from QEMU
Andrew Boie6acbe632015-07-17 12:03:52 -0700556 log_out_fp.write(line)
557 log_out_fp.flush()
558 line = line.strip()
559 verbose("QEMU: %s" % line)
560
Anas Nashif576be982017-12-23 20:20:27 -0500561 harness.handle(line)
562 if harness.state:
Anas Nashif39ae72b2018-08-29 01:45:38 -0400563 # if we have registered a fail make sure the state is not
564 # overridden by a false success message coming from the
565 # testsuite
566 if out_state != 'failed':
567 out_state = harness.state
568
569 # if we get some state, that means test is doing well, we reset
570 # the timeout and wait for 5 more seconds just in case we have
571 # crashed after test has completed
572
573 if harness.type:
574 break
575 else:
576 timeout_time = time.time() + 2
Andrew Boie6acbe632015-07-17 12:03:52 -0700577
578 # TODO: Add support for getting numerical performance data
579 # from test cases. Will involve extending test case reporting
580 # APIs. Add whatever gets reported to the metrics dictionary
581 line = ""
582
Anas Nashifc8390f12017-11-25 17:14:12 -0500583 metrics["handler_time"] = time.time() - start_time
Andrew Boie6acbe632015-07-17 12:03:52 -0700584 verbose("QEMU complete (%s) after %f seconds" %
Anas Nashifc8390f12017-11-25 17:14:12 -0500585 (out_state, metrics["handler_time"]))
Andrew Boie6acbe632015-07-17 12:03:52 -0700586 handler.set_state(out_state, metrics)
587
588 log_out_fp.close()
589 out_fp.close()
590 in_fp.close()
591
592 pid = int(open(pid_fn).read())
593 os.unlink(pid_fn)
Andrew Boie08ce5a52016-02-22 13:28:10 -0800594 try:
595 os.kill(pid, signal.SIGTERM)
596 except ProcessLookupError:
597 # Oh well, as long as it's dead! User probably sent Ctrl-C
598 pass
599
Andrew Boie6acbe632015-07-17 12:03:52 -0700600 os.unlink(fifo_in)
601 os.unlink(fifo_out)
602
Anas Nashif576be982017-12-23 20:20:27 -0500603 def __init__(self, instance):
Andrew Boie6acbe632015-07-17 12:03:52 -0700604 """Constructor
605
Anas Nashifd3384fb2018-02-22 06:44:16 -0600606 @param instance Test instance
Andrew Boie6acbe632015-07-17 12:03:52 -0700607 """
Anas Nashif576be982017-12-23 20:20:27 -0500608
Anas Nashif576be982017-12-23 20:20:27 -0500609 super().__init__(instance)
Anas Nashif576be982017-12-23 20:20:27 -0500610
Andrew Boie6acbe632015-07-17 12:03:52 -0700611 self.results = {}
Anas Nashif13773752018-07-06 18:20:23 -0500612 self.run = True
Andrew Boie6acbe632015-07-17 12:03:52 -0700613
614 # We pass this to QEMU which looks for fifos with .in and .out
615 # suffixes.
Anas Nashif576be982017-12-23 20:20:27 -0500616 self.fifo_fn = os.path.join(instance.outdir, "qemu-fifo")
Andrew Boie6acbe632015-07-17 12:03:52 -0700617
Anas Nashif576be982017-12-23 20:20:27 -0500618 self.pid_fn = os.path.join(instance.outdir, "qemu.pid")
Andrew Boie6acbe632015-07-17 12:03:52 -0700619 if os.path.exists(self.pid_fn):
620 os.unlink(self.pid_fn)
621
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500622 self.log_fn = self.log
Anas Nashif576be982017-12-23 20:20:27 -0500623
624 harness_import = HarnessImporter(instance.test.harness.capitalize())
625 harness = harness_import.instance
Anas Nashifd3384fb2018-02-22 06:44:16 -0600626 harness.configure(self.instance)
627 self.thread = threading.Thread(name=self.name, target=QEMUHandler._thread,
628 args=(self, self.timeout, self.outdir,
Jaakko Hannikainenca505f82016-08-22 15:03:46 +0300629 self.log_fn, self.fifo_fn,
Anas Nashif576be982017-12-23 20:20:27 -0500630 self.pid_fn, self.results, harness))
Anas Nashife0a6a0b2018-02-15 20:07:24 -0600631
632 self.instance.results = harness.tests
Andrew Boie6acbe632015-07-17 12:03:52 -0700633 self.thread.daemon = True
Anas Nashifd3384fb2018-02-22 06:44:16 -0600634 verbose("Spawning QEMU process for %s" % self.name)
Andrew Boie6acbe632015-07-17 12:03:52 -0700635 self.thread.start()
636
Andrew Boie6acbe632015-07-17 12:03:52 -0700637 def get_fifo(self):
638 return self.fifo_fn
639
Anas Nashif3ba1d432017-12-05 15:28:44 -0500640
Andrew Boie6acbe632015-07-17 12:03:52 -0700641class SizeCalculator:
Andrew Boie73b4ee62015-10-07 11:33:22 -0700642
Erwin Rolcb3d1272018-02-10 11:40:40 +0100643 alloc_sections = ["bss", "noinit", "app_bss", "app_noinit", "ccm_bss",
644 "ccm_noinit"]
Andrew Boie18ba1532017-01-17 13:47:06 -0800645 rw_sections = ["datas", "initlevel", "_k_task_list", "_k_event_list",
Allan Stephens3f5c74c2016-11-01 14:37:15 -0500646 "_k_memory_pool", "exceptions", "initshell",
Andrew Boie3ef0b562017-08-31 12:36:45 -0700647 "_static_thread_area", "_k_timer_area", "_k_work_area",
648 "_k_mem_slab_area", "_k_mem_pool_area",
649 "_k_sem_area", "_k_mutex_area", "_k_alert_area",
650 "_k_fifo_area", "_k_lifo_area", "_k_stack_area",
651 "_k_msgq_area", "_k_mbox_area", "_k_pipe_area",
Jukka Rissanen60492072018-02-07 15:00:08 +0200652 "net_if", "net_if_dev", "net_stack", "net_l2_data",
Andrew Boie945af952017-08-22 13:15:23 -0700653 "_k_queue_area", "_net_buf_pool_area", "app_datas",
Erwin Rolcb3d1272018-02-10 11:40:40 +0100654 "kobject_data", "mmu_tables", "app_pad", "priv_stacks",
Anas Nashif0b685602018-06-29 12:57:47 -0500655 "ccm_data", "usb_descriptor", "usb_data", "usb_bos_desc",
656 'log_backends_sections', 'log_dynamic_sections',
Shawn Mosley573f32b2018-04-26 10:14:02 -0400657 'log_const_sections',"app_smem"]
Andrew Boie73b4ee62015-10-07 11:33:22 -0700658 # These get copied into RAM only on non-XIP
Andrew Boie877f82e2017-10-17 11:20:22 -0700659 ro_sections = ["text", "ctors", "init_array", "reset", "object_access",
Andy Rossa3a7e8e2018-05-23 16:39:16 -0700660 "rodata", "devconfig", "net_l2", "vector", "_bt_settings_area"]
Andrew Boie73b4ee62015-10-07 11:33:22 -0700661
Andrew Boie52fef672016-11-29 12:21:59 -0800662 def __init__(self, filename, extra_sections):
Andrew Boie6acbe632015-07-17 12:03:52 -0700663 """Constructor
664
Andrew Boiebbd670c2015-08-17 13:16:11 -0700665 @param filename Path to the output binary
666 The <filename> is parsed by objdump to determine section sizes
Andrew Boie6acbe632015-07-17 12:03:52 -0700667 """
Andrew Boie6acbe632015-07-17 12:03:52 -0700668 # Make sure this is an ELF binary
Andrew Boiebbd670c2015-08-17 13:16:11 -0700669 with open(filename, "rb") as f:
Andrew Boie6acbe632015-07-17 12:03:52 -0700670 magic = f.read(4)
671
Anas Nashifb4bdd662018-08-15 17:12:28 -0500672 try:
673 if (magic != b'\x7fELF'):
674 raise SanityRuntimeError("%s is not an ELF binary" % filename)
675 except Exception as e:
676 print(str(e))
677 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -0700678
679 # Search for CONFIG_XIP in the ELF's list of symbols using NM and AWK.
Anas Nashif3ba1d432017-12-05 15:28:44 -0500680 # GREP can not be used as it returns an error if the symbol is not
681 # found.
682 is_xip_command = "nm " + filename + \
683 " | awk '/CONFIG_XIP/ { print $3 }'"
684 is_xip_output = subprocess.check_output(
685 is_xip_command, shell=True, stderr=subprocess.STDOUT).decode(
686 "utf-8").strip()
Anas Nashifb4bdd662018-08-15 17:12:28 -0500687 try:
688 if is_xip_output.endswith("no symbols"):
689 raise SanityRuntimeError("%s has no symbol information" % filename)
690 except Exception as e:
691 print(str(e))
692 sys.exit(2)
693
Andrew Boie6acbe632015-07-17 12:03:52 -0700694 self.is_xip = (len(is_xip_output) != 0)
695
Andrew Boiebbd670c2015-08-17 13:16:11 -0700696 self.filename = filename
Andrew Boie73b4ee62015-10-07 11:33:22 -0700697 self.sections = []
698 self.rom_size = 0
Andrew Boie6acbe632015-07-17 12:03:52 -0700699 self.ram_size = 0
Andrew Boie52fef672016-11-29 12:21:59 -0800700 self.extra_sections = extra_sections
Andrew Boie6acbe632015-07-17 12:03:52 -0700701
702 self._calculate_sizes()
703
704 def get_ram_size(self):
705 """Get the amount of RAM the application will use up on the device
706
707 @return amount of RAM, in bytes
708 """
Andrew Boie73b4ee62015-10-07 11:33:22 -0700709 return self.ram_size
Andrew Boie6acbe632015-07-17 12:03:52 -0700710
711 def get_rom_size(self):
712 """Get the size of the data that this application uses on device's flash
713
714 @return amount of ROM, in bytes
715 """
Andrew Boie73b4ee62015-10-07 11:33:22 -0700716 return self.rom_size
Andrew Boie6acbe632015-07-17 12:03:52 -0700717
718 def unrecognized_sections(self):
719 """Get a list of sections inside the binary that weren't recognized
720
David B. Kinder29963c32017-06-16 12:32:42 -0700721 @return list of unrecognized section names
Andrew Boie6acbe632015-07-17 12:03:52 -0700722 """
723 slist = []
Andrew Boie73b4ee62015-10-07 11:33:22 -0700724 for v in self.sections:
Andrew Boie6acbe632015-07-17 12:03:52 -0700725 if not v["recognized"]:
Andrew Boie73b4ee62015-10-07 11:33:22 -0700726 slist.append(v["name"])
Andrew Boie6acbe632015-07-17 12:03:52 -0700727 return slist
728
729 def _calculate_sizes(self):
730 """ Calculate RAM and ROM usage by section """
Andrew Boiebbd670c2015-08-17 13:16:11 -0700731 objdump_command = "objdump -h " + self.filename
Anas Nashif3ba1d432017-12-05 15:28:44 -0500732 objdump_output = subprocess.check_output(
733 objdump_command, shell=True).decode("utf-8").splitlines()
Andrew Boie6acbe632015-07-17 12:03:52 -0700734
735 for line in objdump_output:
736 words = line.split()
737
738 if (len(words) == 0): # Skip lines that are too short
739 continue
740
741 index = words[0]
742 if (not index[0].isdigit()): # Skip lines that do not start
743 continue # with a digit
744
745 name = words[1] # Skip lines with section names
746 if (name[0] == '.'): # starting with '.'
747 continue
748
Andrew Boie73b4ee62015-10-07 11:33:22 -0700749 # TODO this doesn't actually reflect the size in flash or RAM as
750 # it doesn't include linker-imposed padding between sections.
751 # It is close though.
Andrew Boie6acbe632015-07-17 12:03:52 -0700752 size = int(words[2], 16)
Andrew Boie9882dcd2015-10-07 14:25:51 -0700753 if size == 0:
754 continue
755
Andrew Boie73b4ee62015-10-07 11:33:22 -0700756 load_addr = int(words[4], 16)
757 virt_addr = int(words[3], 16)
Andrew Boie6acbe632015-07-17 12:03:52 -0700758
759 # Add section to memory use totals (for both non-XIP and XIP scenarios)
Andrew Boie6acbe632015-07-17 12:03:52 -0700760 # Unrecognized section names are not included in the calculations.
Andrew Boie6acbe632015-07-17 12:03:52 -0700761 recognized = True
Andrew Boie73b4ee62015-10-07 11:33:22 -0700762 if name in SizeCalculator.alloc_sections:
763 self.ram_size += size
764 stype = "alloc"
765 elif name in SizeCalculator.rw_sections:
766 self.ram_size += size
767 self.rom_size += size
768 stype = "rw"
769 elif name in SizeCalculator.ro_sections:
770 self.rom_size += size
771 if not self.is_xip:
772 self.ram_size += size
773 stype = "ro"
Andrew Boie6acbe632015-07-17 12:03:52 -0700774 else:
Andrew Boie73b4ee62015-10-07 11:33:22 -0700775 stype = "unknown"
Andrew Boie52fef672016-11-29 12:21:59 -0800776 if name not in self.extra_sections:
777 recognized = False
Andrew Boie6acbe632015-07-17 12:03:52 -0700778
Anas Nashif3ba1d432017-12-05 15:28:44 -0500779 self.sections.append({"name": name, "load_addr": load_addr,
780 "size": size, "virt_addr": virt_addr,
781 "type": stype, "recognized": recognized})
Andrew Boie6acbe632015-07-17 12:03:52 -0700782
783
784class MakeGoal:
785 """Metadata class representing one of the sub-makes called by MakeGenerator
786
David B. Kinder29963c32017-06-16 12:32:42 -0700787 MakeGenerator returns a dictionary of these which can then be associated
Andrew Boie6acbe632015-07-17 12:03:52 -0700788 with TestInstances to get a complete picture of what happened during a test.
789 MakeGenerator is used for tasks outside of building tests (such as
790 defconfigs) which is why MakeGoal is a separate class from TestInstance.
791 """
Anas Nashif3ba1d432017-12-05 15:28:44 -0500792
Anas Nashif4d25b502017-11-25 17:37:17 -0500793 def __init__(self, name, text, handler, make_log, build_log, run_log, handler_log):
Andrew Boie6acbe632015-07-17 12:03:52 -0700794 self.name = name
795 self.text = text
Anas Nashif4d25b502017-11-25 17:37:17 -0500796 self.handler = handler
Andrew Boie6acbe632015-07-17 12:03:52 -0700797 self.make_log = make_log
798 self.build_log = build_log
799 self.run_log = run_log
Anas Nashif4d25b502017-11-25 17:37:17 -0500800 self.handler_log = handler_log
Andrew Boie6acbe632015-07-17 12:03:52 -0700801 self.make_state = "waiting"
802 self.failed = False
803 self.finished = False
804 self.reason = None
805 self.metrics = {}
806
807 def get_error_log(self):
808 if self.make_state == "waiting":
809 # Shouldn't ever see this; breakage in the main Makefile itself.
810 return self.make_log
811 elif self.make_state == "building":
812 # Failure when calling the sub-make to build the code
813 return self.build_log
814 elif self.make_state == "running":
Anas Nashif3ac7b3a2017-08-03 10:03:02 -0400815 # Failure in sub-make for "make run", qemu probably failed to start
Andrew Boie6acbe632015-07-17 12:03:52 -0700816 return self.run_log
817 elif self.make_state == "finished":
Anas Nashifcc164222017-12-26 11:02:46 -0500818 # Execution handler finished, but timed out or otherwise wasn't successful
Anas Nashif4d25b502017-11-25 17:37:17 -0500819 return self.handler_log
Andrew Boie6acbe632015-07-17 12:03:52 -0700820
821 def fail(self, reason):
822 self.failed = True
823 self.finished = True
824 self.reason = reason
825
826 def success(self):
827 self.finished = True
828
829 def __str__(self):
830 if self.finished:
831 if self.failed:
832 return "[%s] failed (%s: see %s)" % (self.name, self.reason,
833 self.get_error_log())
834 else:
835 return "[%s] passed" % self.name
836 else:
837 return "[%s] in progress (%s)" % (self.name, self.make_state)
838
839
840class MakeGenerator:
841 """Generates a Makefile which just calls a bunch of sub-make sessions
842
843 In any given test suite we may need to build dozens if not hundreds of
844 test cases. The cleanest way to parallelize this is to just let Make
845 do the parallelization, sharing the jobserver among all the different
846 sub-make targets.
847 """
848
849 GOAL_HEADER_TMPL = """.PHONY: {goal}
850{goal}:
851"""
852
853 MAKE_RULE_TMPL = """\t@echo sanity_test_{phase} {goal} >&2
Anas Nashiffb91ad62017-10-31 08:33:17 -0400854\tcmake \\
Anas Nashifa8a13882017-12-30 13:01:06 -0500855\t\t-G"{generator}"\\
Anas Nashiffb91ad62017-10-31 08:33:17 -0400856\t\t-H{directory}\\
857\t\t-B{outdir}\\
858\t\t-DEXTRA_CFLAGS="-Werror {cflags}"\\
859\t\t-DEXTRA_AFLAGS=-Wa,--fatal-warnings\\
Anas Nashif262e4a42017-12-14 08:42:45 -0500860\t\t-DEXTRA_LDFLAGS="{ldflags}"\\
Anas Nashiffb91ad62017-10-31 08:33:17 -0400861\t\t{args}\\
862\t\t>{logfile} 2>&1
Anas Nashifa8a13882017-12-30 13:01:06 -0500863\t{generator_cmd} -C {outdir}\\
864\t\t{verb} {make_args}\\
Anas Nashiffb91ad62017-10-31 08:33:17 -0400865\t\t>>{logfile} 2>&1
Andrew Boie6acbe632015-07-17 12:03:52 -0700866"""
Anas Nashif20f553f2018-03-23 11:26:41 -0500867 MAKE_RULE_TMPL_RUN = """\t@echo sanity_test_{phase} {goal} >&2
868\t{generator_cmd} -C {outdir}\\
869\t\t{verb} {make_args}\\
870\t\t>>{logfile} 2>&1
871"""
Andrew Boie6acbe632015-07-17 12:03:52 -0700872
873 GOAL_FOOTER_TMPL = "\t@echo sanity_test_finished {goal} >&2\n\n"
874
Anas Nashif3ba1d432017-12-05 15:28:44 -0500875 re_make = re.compile(
876 "sanity_test_([A-Za-z0-9]+) (.+)|$|make[:] \*\*\* \[(.+:.+: )?(.+)\] Error.+$")
Andrew Boie6acbe632015-07-17 12:03:52 -0700877
Anas Nashif37f9dc52018-02-23 08:53:46 -0600878 def __init__(self, base_outdir):
Andrew Boie6acbe632015-07-17 12:03:52 -0700879 """MakeGenerator constructor
880
881 @param base_outdir Intended to be the base out directory. A make.log
882 file will be created here which contains the output of the
883 top-level Make session, as well as the dynamic control Makefile
884 @param verbose If true, pass V=1 to all the sub-makes which greatly
885 increases their verbosity
886 """
887 self.goals = {}
888 if not os.path.exists(base_outdir):
889 os.makedirs(base_outdir)
890 self.logfile = os.path.join(base_outdir, "make.log")
891 self.makefile = os.path.join(base_outdir, "Makefile")
Anas Nashif37f9dc52018-02-23 08:53:46 -0600892 self.deprecations = options.error_on_deprecations
Andrew Boie6acbe632015-07-17 12:03:52 -0700893
894 def _get_rule_header(self, name):
895 return MakeGenerator.GOAL_HEADER_TMPL.format(goal=name)
896
Anas Nashif3ba1d432017-12-05 15:28:44 -0500897 def _get_sub_make(self, name, phase, workdir, outdir,
898 logfile, args, make_args=""):
Anas Nashiffb91ad62017-10-31 08:33:17 -0400899 """
900 @param args Arguments given to CMake
901 @param make_args Arguments given to the Makefile generated by CMake
902 """
Anas Nashif3ba1d432017-12-05 15:28:44 -0500903 args = " ".join(["-D{}".format(a) for a in args])
Anas Nashif262e4a42017-12-14 08:42:45 -0500904 ldflags = ""
Andrew Boie29599f62018-05-24 13:33:09 -0700905 cflags = ""
Anas Nashife3febe92016-11-30 14:25:44 -0500906
907 if self.deprecations:
908 cflags = cflags + " -Wno-deprecated-declarations"
Andrew Boief7a6e282017-02-02 13:04:57 -0800909
Alberto Escolar Piedras770178b2018-05-02 13:49:51 +0200910 ldflags="-Wl,--fatal-warnings"
Anas Nashif262e4a42017-12-14 08:42:45 -0500911
Anas Nashif25f6ab62018-03-06 07:15:11 -0600912 if options.ninja:
Sebastian Bøe0e6689d2018-01-18 14:40:07 +0100913 generator = "Ninja"
Andy Rossdec163f2018-05-21 10:12:59 -0700914 generator_cmd = "ninja -j1"
Sebastian Bøe0e6689d2018-01-18 14:40:07 +0100915 verb = "-v" if VERBOSE else ""
Anas Nashif25f6ab62018-03-06 07:15:11 -0600916 else:
917 generator = "Unix Makefiles"
918 generator_cmd = "$(MAKE)"
Andrew Boie3efd2692018-06-26 10:41:35 -0700919 verb = "VERBOSE=1" if VERBOSE else ""
Anas Nashifa8a13882017-12-30 13:01:06 -0500920
Anas Nashif20f553f2018-03-23 11:26:41 -0500921 if phase == 'running':
922 return MakeGenerator.MAKE_RULE_TMPL_RUN.format(
923 generator_cmd=generator_cmd,
924 phase=phase,
925 goal=name,
926 outdir=outdir,
927 verb=verb,
928 logfile=logfile,
929 make_args=make_args
930 )
931 else:
932 return MakeGenerator.MAKE_RULE_TMPL.format(
933 generator=generator,
934 generator_cmd=generator_cmd,
935 phase=phase,
936 goal=name,
937 outdir=outdir,
938 cflags=cflags,
939 ldflags=ldflags,
940 directory=workdir,
941 verb=verb,
942 args=args,
943 logfile=logfile,
944 make_args=make_args
945 )
Andrew Boie6acbe632015-07-17 12:03:52 -0700946
947 def _get_rule_footer(self, name):
948 return MakeGenerator.GOAL_FOOTER_TMPL.format(goal=name)
949
Anas Nashif3ba1d432017-12-05 15:28:44 -0500950 def add_build_goal(self, name, directory, outdir,
951 args, buildlog, make_args=""):
Anas Nashif13773752018-07-06 18:20:23 -0500952 """Add a goal to invoke a build session
Andrew Boie6acbe632015-07-17 12:03:52 -0700953
954 @param name A unique string name for this build goal. The results
955 dictionary returned by execute() will be keyed by this name.
956 @param directory Absolute path to working directory, will be passed
957 to make -C
958 @param outdir Absolute path to output directory, will be passed to
Sebastian Bøe71d7de02017-11-09 12:06:04 +0100959 cmake via -B=<path>
960 @param args Extra command line arguments to pass to 'cmake', typically
Andrew Boie6acbe632015-07-17 12:03:52 -0700961 environment variables or specific Make goals
962 """
Andrew Boie6acbe632015-07-17 12:03:52 -0700963
Anas Nashif13773752018-07-06 18:20:23 -0500964 if not os.path.exists(outdir):
965 os.makedirs(outdir)
966
967 build_logfile = os.path.join(outdir, buildlog)
968 text = self._get_rule_header(name)
969 text += self._get_sub_make(name, "building", directory, outdir, build_logfile,
970 args, make_args=make_args)
971 text += self._get_rule_footer(name)
972
973 self.goals[name] = MakeGoal( name, text, None, self.logfile, build_logfile, None, None)
974
975 def add_goal(self, instance, type, args, make_args=""):
Anas Nashif4a9f3e62018-07-06 18:58:18 -0500976
Anas Nashif13773752018-07-06 18:20:23 -0500977 """Add a goal to build a Zephyr project and then run it using a handler
Andrew Boie6acbe632015-07-17 12:03:52 -0700978
979 The generated make goal invokes Make twice, the first time it will
980 build the default goal, and the second will invoke the 'qemu' goal.
Anas Nashif13773752018-07-06 18:20:23 -0500981 The output of the handler session will be monitored, and terminated
Andrew Boie6acbe632015-07-17 12:03:52 -0700982 either upon pass/fail result of the test program, or the timeout
983 is reached.
984
Sebastian Bøe71d7de02017-11-09 12:06:04 +0100985 @param args Extra cache entries to define in CMake.
Andrew Boie6acbe632015-07-17 12:03:52 -0700986 """
987
Anas Nashif576be982017-12-23 20:20:27 -0500988 name = instance.name
989 directory = instance.test.code_location
990 outdir = instance.outdir
991
Andrew Boie6acbe632015-07-17 12:03:52 -0700992 build_logfile = os.path.join(outdir, "build.log")
993 run_logfile = os.path.join(outdir, "run.log")
Andrew Boie6acbe632015-07-17 12:03:52 -0700994
Anas Nashif13773752018-07-06 18:20:23 -0500995 if not os.path.exists(outdir):
996 os.makedirs(outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -0700997
Anas Nashif13773752018-07-06 18:20:23 -0500998 handler = None
999 if type == "qemu":
1000 handler = QEMUHandler(instance)
1001 elif type == "native":
Anas Nashifdf7ee612018-07-07 06:09:01 -05001002 handler = BinaryHandler(instance)
1003 handler.binary = os.path.join(outdir, "zephyr", "zephyr.exe")
Anas Nashif99f5a6c2018-07-07 08:45:53 -05001004 elif type == "nsim":
1005 handler = BinaryHandler(instance)
1006 handler.call_make_run = True
Anas Nashif13773752018-07-06 18:20:23 -05001007 elif type == "unit":
Anas Nashifdf7ee612018-07-07 06:09:01 -05001008 handler = BinaryHandler(instance)
1009 handler.binary = os.path.join(outdir, "testbinary")
Anas Nashif13773752018-07-06 18:20:23 -05001010 elif type == "device":
1011 handler = DeviceHandler(instance)
Anas Nashif576be982017-12-23 20:20:27 -05001012
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02001013 if options.enable_coverage:
1014 args += ["COVERAGE=1", "EXTRA_LDFLAGS=--coverage"]
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02001015 args += ["CONFIG_COVERAGE=y"]
1016
Anas Nashif13773752018-07-06 18:20:23 -05001017 if type == 'qemu':
1018 args.append("QEMU_PIPE=%s" % handler.get_fifo())
1019
Anas Nashifcc164222017-12-26 11:02:46 -05001020 text = (self._get_rule_header(name) +
1021 self._get_sub_make(name, "building", directory,
Anas Nashif13773752018-07-06 18:20:23 -05001022 outdir, build_logfile, args, make_args=make_args))
1023 if handler and handler.run:
1024 text += self._get_sub_make(name, "running", directory,
1025 outdir, run_logfile,
1026 args, make_args="run")
Anas Nashif4d25b502017-11-25 17:37:17 -05001027
Anas Nashif13773752018-07-06 18:20:23 -05001028 text += self._get_rule_footer(name)
Anas Nashif73440ea2018-02-19 10:57:03 -06001029
Anas Nashif73440ea2018-02-19 10:57:03 -06001030 self.goals[name] = MakeGoal(name, text, handler, self.logfile, build_logfile,
Anas Nashif4a9f3e62018-07-06 18:58:18 -05001031 run_logfile, handler.log if handler else None)
Anas Nashif73440ea2018-02-19 10:57:03 -06001032
Anas Nashif13773752018-07-06 18:20:23 -05001033
Anas Nashif37f9dc52018-02-23 08:53:46 -06001034 def add_test_instance(self, ti, extra_args=[]):
Andrew Boie6acbe632015-07-17 12:03:52 -07001035 """Add a goal to build/test a TestInstance object
1036
1037 @param ti TestInstance object to build. The status dictionary returned
1038 by execute() will be keyed by its .name field.
1039 """
1040 args = ti.test.extra_args[:]
Anas Nashiffa695d22017-10-04 16:14:27 -04001041 if len(ti.test.extra_configs) > 0:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001042 args.append("OVERLAY_CONFIG=%s" %
1043 os.path.join(ti.outdir, "overlay.conf"))
Anas Nashiffa695d22017-10-04 16:14:27 -04001044
Anas Nashiffb91ad62017-10-31 08:33:17 -04001045 args.append("BOARD={}".format(ti.platform.name))
Andrew Boieba612002016-09-01 10:41:03 -07001046 args.extend(extra_args)
Anas Nashif5df8cff2018-02-23 08:37:14 -06001047
Anas Nashif37f9dc52018-02-23 08:53:46 -06001048 do_run_slow = options.enable_slow or not ti.test.slow
1049 do_build_only = ti.build_only or options.build_only
Anas Nashif5df8cff2018-02-23 08:37:14 -06001050 do_run = (not do_build_only) and do_run_slow
1051
Anas Nashif13773752018-07-06 18:20:23 -05001052 # FIXME: Need refactoring and cleanup
1053 type = None
Anas Nashif5df8cff2018-02-23 08:37:14 -06001054 if ti.platform.qemu_support and do_run:
Anas Nashif13773752018-07-06 18:20:23 -05001055 type = "qemu"
Jaakko Hannikainenca505f82016-08-22 15:03:46 +03001056 elif ti.test.type == "unit":
Anas Nashif13773752018-07-06 18:20:23 -05001057 type = "unit"
Anas Nashif5df8cff2018-02-23 08:37:14 -06001058 elif ti.platform.type == "native" and do_run:
Anas Nashif13773752018-07-06 18:20:23 -05001059 type = "native"
Anas Nashif99f5a6c2018-07-07 08:45:53 -05001060 elif ti.platform.simulation == "nsim" and do_run:
Anas Nashife24350c2018-07-11 15:09:22 -05001061 if find_executable("nsimdrv"):
1062 type = "nsim"
Anas Nashif37f9dc52018-02-23 08:53:46 -06001063 elif options.device_testing and (not ti.build_only) and (not options.build_only):
Anas Nashif13773752018-07-06 18:20:23 -05001064 type = "device"
Anas Nashif37f9dc52018-02-23 08:53:46 -06001065
Anas Nashif13773752018-07-06 18:20:23 -05001066 self.add_goal(ti, type, args)
Andrew Boie6acbe632015-07-17 12:03:52 -07001067
1068 def execute(self, callback_fn=None, context=None):
1069 """Execute all the registered build goals
1070
1071 @param callback_fn If not None, a callback function will be called
1072 as individual goals transition between states. This function
1073 should accept two parameters: a string state and an arbitrary
1074 context object, supplied here
1075 @param context Context object to pass to the callback function.
1076 Type and semantics are specific to that callback function.
1077 @return A dictionary mapping goal names to final status.
1078 """
1079
Andrew Boie08ce5a52016-02-22 13:28:10 -08001080 with open(self.makefile, "wt") as tf, \
Andrew Boie6acbe632015-07-17 12:03:52 -07001081 open(os.devnull, "wb") as devnull, \
Andrew Boie08ce5a52016-02-22 13:28:10 -08001082 open(self.logfile, "wt") as make_log:
Andrew Boie6acbe632015-07-17 12:03:52 -07001083 # Create our dynamic Makefile and execute it.
1084 # Watch stderr output which is where we will keep
1085 # track of build state
Andrew Boie08ce5a52016-02-22 13:28:10 -08001086 for name, goal in self.goals.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001087 tf.write(goal.text)
1088 tf.write("all: %s\n" % (" ".join(self.goals.keys())))
1089 tf.flush()
1090
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03001091 cmd = ["make", "-k", "-j", str(JOBS), "-f", tf.name, "all"]
Andy Rossdec163f2018-05-21 10:12:59 -07001092
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03001093 p = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=devnull)
Andrew Boie6acbe632015-07-17 12:03:52 -07001094
1095 for line in iter(p.stderr.readline, b''):
Andrew Boie08ce5a52016-02-22 13:28:10 -08001096 line = line.decode("utf-8")
Andrew Boie6acbe632015-07-17 12:03:52 -07001097 make_log.write(line)
1098 verbose("MAKE: " + repr(line.strip()))
1099 m = MakeGenerator.re_make.match(line)
1100 if not m:
1101 continue
1102
Jaakko Hannikainenca505f82016-08-22 15:03:46 +03001103 state, name, _, error = m.groups()
Andrew Boie6acbe632015-07-17 12:03:52 -07001104 if error:
1105 goal = self.goals[error]
Andrew Boie822b0872017-01-10 13:32:40 -08001106 # Sometimes QEMU will run an image and then crash out, which
Anas Nashif3ac7b3a2017-08-03 10:03:02 -04001107 # will cause the 'make run' invocation to exit with
Andrew Boie822b0872017-01-10 13:32:40 -08001108 # nonzero status.
1109 # Need to distinguish this case from a compilation failure.
Anas Nashif4d25b502017-11-25 17:37:17 -05001110 if goal.handler:
Anas Nashif9a839df2018-01-29 08:42:38 -05001111 goal.fail("handler_crash")
Andrew Boie822b0872017-01-10 13:32:40 -08001112 else:
1113 goal.fail("build_error")
Andrew Boie6acbe632015-07-17 12:03:52 -07001114 else:
Anas Nashifcc164222017-12-26 11:02:46 -05001115 goal = self.goals[name]
1116 goal.make_state = state
1117
Andrew Boie6acbe632015-07-17 12:03:52 -07001118 if state == "finished":
Anas Nashif4d25b502017-11-25 17:37:17 -05001119 if goal.handler:
Anas Nashif576be982017-12-23 20:20:27 -05001120 if hasattr(goal.handler, "handle"):
Anas Nashif4d25b502017-11-25 17:37:17 -05001121 goal.handler.handle()
Anas Nashif4a9f3e62018-07-06 18:58:18 -05001122 goal.handler_log = goal.handler.log
Anas Nashifcc164222017-12-26 11:02:46 -05001123
Anas Nashif4d25b502017-11-25 17:37:17 -05001124 thread_status, metrics = goal.handler.get_state()
Andrew Boie6acbe632015-07-17 12:03:52 -07001125 goal.metrics.update(metrics)
1126 if thread_status == "passed":
1127 goal.success()
1128 else:
1129 goal.fail(thread_status)
1130 else:
1131 goal.success()
1132
1133 if callback_fn:
1134 callback_fn(context, self.goals, goal)
1135
1136 p.wait()
1137 return self.goals
1138
1139
1140# "list" - List of strings
1141# "list:<type>" - List of <type>
1142# "set" - Set of unordered, unique strings
1143# "set:<type>" - Set of <type>
1144# "float" - Floating point
1145# "int" - Integer
1146# "bool" - Boolean
1147# "str" - String
1148
1149# XXX Be sure to update __doc__ if you change any of this!!
1150
Anas Nashif3ba1d432017-12-05 15:28:44 -05001151platform_valid_keys = {"qemu_support": {"type": "bool", "default": False},
1152 "supported_toolchains": {"type": "list", "default": []}}
Andrew Boie6acbe632015-07-17 12:03:52 -07001153
Anas Nashif3ba1d432017-12-05 15:28:44 -05001154testcase_valid_keys = {"tags": {"type": "set", "required": False},
1155 "type": {"type": "str", "default": "integration"},
1156 "extra_args": {"type": "list"},
1157 "extra_configs": {"type": "list"},
1158 "build_only": {"type": "bool", "default": False},
1159 "build_on_all": {"type": "bool", "default": False},
1160 "skip": {"type": "bool", "default": False},
1161 "slow": {"type": "bool", "default": False},
1162 "timeout": {"type": "int", "default": 60},
1163 "min_ram": {"type": "int", "default": 8},
1164 "depends_on": {"type": "set"},
1165 "min_flash": {"type": "int", "default": 32},
1166 "arch_whitelist": {"type": "set"},
1167 "arch_exclude": {"type": "set"},
1168 "extra_sections": {"type": "list", "default": []},
1169 "platform_exclude": {"type": "set"},
1170 "platform_whitelist": {"type": "set"},
1171 "toolchain_exclude": {"type": "set"},
1172 "toolchain_whitelist": {"type": "set"},
Anas Nashifab940162017-12-08 10:17:57 -05001173 "filter": {"type": "str"},
Anas Nashif576be982017-12-23 20:20:27 -05001174 "harness": {"type": "str"},
1175 "harness_config": {"type": "map"}
Anas Nashifab940162017-12-08 10:17:57 -05001176 }
Andrew Boie6acbe632015-07-17 12:03:52 -07001177
1178
1179class SanityConfigParser:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001180 """Class to read test case files with semantic checking
Andrew Boie6acbe632015-07-17 12:03:52 -07001181 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001182
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001183 def __init__(self, filename, schema):
Andrew Boie6acbe632015-07-17 12:03:52 -07001184 """Instantiate a new SanityConfigParser object
1185
Anas Nashifa792a3d2017-04-04 18:47:49 -04001186 @param filename Source .yaml file to read
Andrew Boie6acbe632015-07-17 12:03:52 -07001187 """
Anas Nashif255625b2017-12-05 15:08:26 -05001188 self.data = scl.yaml_load_verify(filename, schema)
Andrew Boie6acbe632015-07-17 12:03:52 -07001189 self.filename = filename
Anas Nashif255625b2017-12-05 15:08:26 -05001190 self.tests = {}
1191 self.common = {}
1192 if 'tests' in self.data:
1193 self.tests = self.data['tests']
1194 if 'common' in self.data:
1195 self.common = self.data['common']
Andrew Boie6acbe632015-07-17 12:03:52 -07001196
1197 def _cast_value(self, value, typestr):
Anas Nashif3ba1d432017-12-05 15:28:44 -05001198 if isinstance(value, str):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001199 v = value.strip()
Andrew Boie6acbe632015-07-17 12:03:52 -07001200 if typestr == "str":
1201 return v
1202
1203 elif typestr == "float":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001204 return float(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001205
1206 elif typestr == "int":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001207 return int(value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001208
1209 elif typestr == "bool":
Anas Nashifa792a3d2017-04-04 18:47:49 -04001210 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001211
Anas Nashif3ba1d432017-12-05 15:28:44 -05001212 elif typestr.startswith("list") and isinstance(value, list):
Anas Nashiffa695d22017-10-04 16:14:27 -04001213 return value
Anas Nashif3ba1d432017-12-05 15:28:44 -05001214 elif typestr.startswith("list") and isinstance(value, str):
Andrew Boie6acbe632015-07-17 12:03:52 -07001215 vs = v.split()
1216 if len(typestr) > 4 and typestr[4] == ":":
1217 return [self._cast_value(vsi, typestr[5:]) for vsi in vs]
1218 else:
1219 return vs
1220
1221 elif typestr.startswith("set"):
1222 vs = v.split()
1223 if len(typestr) > 3 and typestr[3] == ":":
1224 return set([self._cast_value(vsi, typestr[4:]) for vsi in vs])
1225 else:
1226 return set(vs)
1227
Anas Nashif576be982017-12-23 20:20:27 -05001228 elif typestr.startswith("map"):
1229 return value
Andrew Boie6acbe632015-07-17 12:03:52 -07001230 else:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001231 raise ConfigurationError(
1232 self.filename, "unknown type '%s'" % value)
Andrew Boie6acbe632015-07-17 12:03:52 -07001233
Anas Nashifb4754ed2017-12-05 17:27:58 -05001234 def get_test(self, name, valid_keys):
1235 """Get a dictionary representing the keys/values within a test
Andrew Boie6acbe632015-07-17 12:03:52 -07001236
Anas Nashifb4754ed2017-12-05 17:27:58 -05001237 @param name The test in the .yaml file to retrieve data from
Andrew Boie6acbe632015-07-17 12:03:52 -07001238 @param valid_keys A dictionary representing the intended semantics
Anas Nashifb4754ed2017-12-05 17:27:58 -05001239 for this test. Each key in this dictionary is a key that could
Anas Nashifa792a3d2017-04-04 18:47:49 -04001240 be specified, if a key is given in the .yaml file which isn't in
Andrew Boie6acbe632015-07-17 12:03:52 -07001241 here, it will generate an error. Each value in this dictionary
1242 is another dictionary containing metadata:
1243
1244 "default" - Default value if not given
1245 "type" - Data type to convert the text value to. Simple types
1246 supported are "str", "float", "int", "bool" which will get
1247 converted to respective Python data types. "set" and "list"
1248 may also be specified which will split the value by
1249 whitespace (but keep the elements as strings). finally,
1250 "list:<type>" and "set:<type>" may be given which will
1251 perform a type conversion after splitting the value up.
1252 "required" - If true, raise an error if not defined. If false
1253 and "default" isn't specified, a type conversion will be
1254 done on an empty string
Anas Nashifb4754ed2017-12-05 17:27:58 -05001255 @return A dictionary containing the test key-value pairs with
Andrew Boie6acbe632015-07-17 12:03:52 -07001256 type conversion and default values filled in per valid_keys
1257 """
1258
1259 d = {}
Anas Nashif255625b2017-12-05 15:08:26 -05001260 for k, v in self.common.items():
Anas Nashiffa695d22017-10-04 16:14:27 -04001261 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001262
Anas Nashifb4754ed2017-12-05 17:27:58 -05001263 for k, v in self.tests[name].items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001264 if k not in valid_keys:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001265 raise ConfigurationError(
1266 self.filename,
1267 "Unknown config key '%s' in definition for '%s'" %
Anas Nashifb4754ed2017-12-05 17:27:58 -05001268 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001269
Anas Nashiffa695d22017-10-04 16:14:27 -04001270 if k in d:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001271 if isinstance(d[k], str):
Anas Nashiffa695d22017-10-04 16:14:27 -04001272 d[k] += " " + v
1273 else:
1274 d[k] = v
Anas Nashif75547e22018-02-24 08:32:14 -06001275
Andrew Boie08ce5a52016-02-22 13:28:10 -08001276 for k, kinfo in valid_keys.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001277 if k not in d:
1278 if "required" in kinfo:
1279 required = kinfo["required"]
1280 else:
1281 required = False
1282
1283 if required:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001284 raise ConfigurationError(
1285 self.filename,
Anas Nashifb4754ed2017-12-05 17:27:58 -05001286 "missing required value for '%s' in test '%s'" %
1287 (k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001288 else:
1289 if "default" in kinfo:
1290 default = kinfo["default"]
1291 else:
1292 default = self._cast_value("", kinfo["type"])
1293 d[k] = default
1294 else:
1295 try:
1296 d[k] = self._cast_value(d[k], kinfo["type"])
Andrew Boie08ce5a52016-02-22 13:28:10 -08001297 except ValueError as ve:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001298 raise ConfigurationError(
Anas Nashifb4754ed2017-12-05 17:27:58 -05001299 self.filename, "bad %s value '%s' for key '%s' in name '%s'" %
1300 (kinfo["type"], d[k], k, name))
Andrew Boie6acbe632015-07-17 12:03:52 -07001301
1302 return d
1303
1304
1305class Platform:
1306 """Class representing metadata for a particular platform
1307
Anas Nashifc7406082015-12-13 15:00:31 -05001308 Maps directly to BOARD when building"""
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001309
1310 yaml_platform_schema = scl.yaml_load(
Anas Nashif3ba1d432017-12-05 15:28:44 -05001311 os.path.join(
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +03001312 ZEPHYR_BASE,
Anas Nashif3ba1d432017-12-05 15:28:44 -05001313 "scripts",
1314 "sanity_chk",
1315 "sanitycheck-platform-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001316
Anas Nashifa792a3d2017-04-04 18:47:49 -04001317 def __init__(self, cfile):
Andrew Boie6acbe632015-07-17 12:03:52 -07001318 """Constructor.
1319
Anas Nashif877d3ca2017-12-05 17:39:29 -05001320 @param cfile Path to platform configuration file, which gives
1321 info about the platform to be added.
Andrew Boie6acbe632015-07-17 12:03:52 -07001322 """
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001323 scp = SanityConfigParser(cfile, self.yaml_platform_schema)
Anas Nashif255625b2017-12-05 15:08:26 -05001324 data = scp.data
Anas Nashifa792a3d2017-04-04 18:47:49 -04001325
Anas Nashif255625b2017-12-05 15:08:26 -05001326 self.name = data['identifier']
Anas Nashiff3d48e12018-07-24 08:14:42 -05001327 self.sanitycheck = data.get("sanitycheck", True)
Anas Nashifa792a3d2017-04-04 18:47:49 -04001328 # if no RAM size is specified by the board, take a default of 128K
Anas Nashif255625b2017-12-05 15:08:26 -05001329 self.ram = data.get("ram", 128)
1330 testing = data.get("testing", {})
Anas Nashifa792a3d2017-04-04 18:47:49 -04001331 self.ignore_tags = testing.get("ignore_tags", [])
1332 self.default = testing.get("default", False)
1333 # if no flash size is specified by the board, take a default of 512K
Anas Nashif255625b2017-12-05 15:08:26 -05001334 self.flash = data.get("flash", 512)
Anas Nashif95276932017-07-31 08:47:41 -04001335 self.supported = set()
Anas Nashif255625b2017-12-05 15:08:26 -05001336 for supp_feature in data.get("supported", []):
Anas Nashif95276932017-07-31 08:47:41 -04001337 for item in supp_feature.split(":"):
1338 self.supported.add(item)
1339
Anas Nashif8acdbd72018-01-04 14:15:22 -05001340 self.qemu_support = True if data.get('simulation', "na") == 'qemu' else False
Anas Nashif255625b2017-12-05 15:08:26 -05001341 self.arch = data['arch']
Anas Nashifcc164222017-12-26 11:02:46 -05001342 self.type = data.get('type', "na")
Anas Nashif8acdbd72018-01-04 14:15:22 -05001343 self.simulation = data.get('simulation', "na")
Anas Nashif255625b2017-12-05 15:08:26 -05001344 self.supported_toolchains = data.get("toolchain", [])
Andrew Boie41878222016-11-03 11:58:53 -07001345 self.defconfig = None
Andrew Boie6acbe632015-07-17 12:03:52 -07001346 pass
1347
Andrew Boie6acbe632015-07-17 12:03:52 -07001348 def __repr__(self):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001349 return "<%s on %s>" % (self.name, self.arch)
Andrew Boie6acbe632015-07-17 12:03:52 -07001350
1351
1352class Architecture:
1353 """Class representing metadata for a particular architecture
1354 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001355
Anas Nashifa792a3d2017-04-04 18:47:49 -04001356 def __init__(self, name, platforms):
Andrew Boie6acbe632015-07-17 12:03:52 -07001357 """Architecture constructor
1358
Anas Nashif877d3ca2017-12-05 17:39:29 -05001359 @param name String name for this architecture
1360 @param platforms list of platforms belonging to this architecture
Andrew Boie6acbe632015-07-17 12:03:52 -07001361 """
Anas Nashifa792a3d2017-04-04 18:47:49 -04001362 self.platforms = platforms
Andrew Boie6acbe632015-07-17 12:03:52 -07001363
Anas Nashifa792a3d2017-04-04 18:47:49 -04001364 self.name = name
Andrew Boie6acbe632015-07-17 12:03:52 -07001365
1366 def __repr__(self):
1367 return "<arch %s>" % self.name
1368
1369
1370class TestCase:
1371 """Class representing a test application
1372 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001373
Anas Nashif7fae29c2017-10-09 13:19:12 -04001374 def __init__(self, testcase_root, workdir, name, tc_dict, yamlfile):
Andrew Boie6acbe632015-07-17 12:03:52 -07001375 """TestCase constructor.
1376
Anas Nashif877d3ca2017-12-05 17:39:29 -05001377 This gets called by TestSuite as it finds and reads test yaml files.
Anas Nashifa792a3d2017-04-04 18:47:49 -04001378 Multiple TestCase instances may be generated from a single testcase.yaml,
Anas Nashif877d3ca2017-12-05 17:39:29 -05001379 each one corresponds to an entry within that file.
Andrew Boie6acbe632015-07-17 12:03:52 -07001380
Andrew Boie6acbe632015-07-17 12:03:52 -07001381 We need to have a unique name for every single test case. Since
Anas Nashifa792a3d2017-04-04 18:47:49 -04001382 a testcase.yaml can define multiple tests, the canonical name for
Andrew Boie6acbe632015-07-17 12:03:52 -07001383 the test case is <workdir>/<name>.
1384
1385 @param testcase_root Absolute path to the root directory where
1386 all the test cases live
1387 @param workdir Relative path to the project directory for this
1388 test application from the test_case root.
Anas Nashif877d3ca2017-12-05 17:39:29 -05001389 @param name Name of this test case, corresponding to the entry name
Andrew Boie6acbe632015-07-17 12:03:52 -07001390 in the test case configuration file. For many test cases that just
1391 define one test, can be anything and is usually "test". This is
1392 really only used to distinguish between different cases when
Anas Nashifa792a3d2017-04-04 18:47:49 -04001393 the testcase.yaml defines multiple tests
Anas Nashif877d3ca2017-12-05 17:39:29 -05001394 @param tc_dict Dictionary with test values for this test case
Anas Nashifa792a3d2017-04-04 18:47:49 -04001395 from the testcase.yaml file
Andrew Boie6acbe632015-07-17 12:03:52 -07001396 """
1397 self.code_location = os.path.join(testcase_root, workdir)
Anas Nashifaae71d72018-04-21 22:26:48 -05001398 self.id = name
1399 self.cases = []
Jaakko Hannikainenca505f82016-08-22 15:03:46 +03001400 self.type = tc_dict["type"]
Andrew Boie6acbe632015-07-17 12:03:52 -07001401 self.tags = tc_dict["tags"]
1402 self.extra_args = tc_dict["extra_args"]
Anas Nashiffa695d22017-10-04 16:14:27 -04001403 self.extra_configs = tc_dict["extra_configs"]
Andrew Boie6acbe632015-07-17 12:03:52 -07001404 self.arch_whitelist = tc_dict["arch_whitelist"]
Anas Nashif30d13872015-10-05 10:02:45 -04001405 self.arch_exclude = tc_dict["arch_exclude"]
Anas Nashif2bd99bc2015-10-12 13:10:57 -04001406 self.skip = tc_dict["skip"]
Anas Nashif30d13872015-10-05 10:02:45 -04001407 self.platform_exclude = tc_dict["platform_exclude"]
Andrew Boie6acbe632015-07-17 12:03:52 -07001408 self.platform_whitelist = tc_dict["platform_whitelist"]
Anas Nashifb17e1ca2017-06-27 18:05:30 -04001409 self.toolchain_exclude = tc_dict["toolchain_exclude"]
1410 self.toolchain_whitelist = tc_dict["toolchain_whitelist"]
Andrew Boie3ea78922016-03-24 14:46:00 -07001411 self.tc_filter = tc_dict["filter"]
Andrew Boie6acbe632015-07-17 12:03:52 -07001412 self.timeout = tc_dict["timeout"]
Anas Nashifb0f3ae02017-12-08 12:48:39 -05001413 self.harness = tc_dict["harness"]
Anas Nashif576be982017-12-23 20:20:27 -05001414 self.harness_config = tc_dict["harness_config"]
Andrew Boie6acbe632015-07-17 12:03:52 -07001415 self.build_only = tc_dict["build_only"]
Anas Nashifa792a3d2017-04-04 18:47:49 -04001416 self.build_on_all = tc_dict["build_on_all"]
Andrew Boie6bb087c2016-02-10 13:39:00 -08001417 self.slow = tc_dict["slow"]
Anas Nashifa792a3d2017-04-04 18:47:49 -04001418 self.min_ram = tc_dict["min_ram"]
1419 self.depends_on = tc_dict["depends_on"]
1420 self.min_flash = tc_dict["min_flash"]
Andrew Boie52fef672016-11-29 12:21:59 -08001421 self.extra_sections = tc_dict["extra_sections"]
Anas Nashifbd166f42017-09-02 12:32:08 -04001422
Alberto Escolar Piedras2151b862018-01-29 15:09:21 +01001423 self.path = os.path.normpath(os.path.join(os.path.realpath(
1424 testcase_root).replace(os.path.realpath(ZEPHYR_BASE) + "/", ''),
1425 workdir, name))
1426
Anas Nashifaae71d72018-04-21 22:26:48 -05001427
Anas Nashifbd166f42017-09-02 12:32:08 -04001428 self.name = os.path.join(self.path)
Anas Nashif2cf0df02015-10-10 09:29:43 -04001429 self.defconfig = {}
Anas Nashif7fae29c2017-10-09 13:19:12 -04001430 self.yamlfile = yamlfile
Andrew Boie6acbe632015-07-17 12:03:52 -07001431
Anas Nashifaae71d72018-04-21 22:26:48 -05001432 def scan_file(self, inf_name):
Anas Nashifaae71d72018-04-21 22:26:48 -05001433 suite_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001434 # do not match until end-of-line, otherwise we won't allow
1435 # stc_regex below to catch the ones that are declared in the same
1436 # line--as we only search starting the end of this match
1437 br"^\s*ztest_test_suite\(\s*(?P<suite_name>[a-zA-Z0-9_]+)\s*,",
Anas Nashifaae71d72018-04-21 22:26:48 -05001438 re.MULTILINE)
1439 stc_regex = re.compile(
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001440 br"^\s*" # empy space at the beginning is ok
1441 # catch the case where it is declared in the same sentence, e.g:
1442 #
1443 # ztest_test_suite(mutex_complex, ztest_user_unit_test(TESTNAME));
1444 br"(?:ztest_test_suite\([a-zA-Z0-9_]+,\s*)?"
1445 # Catch ztest[_user]_unit_test-[_setup_teardown](TESTNAME)
1446 br"ztest_(?:user_)?unit_test(?:_setup_teardown)?"
1447 # Consume the argument that becomes the extra testcse
1448 br"\(\s*"
1449 br"(?P<stc_name>[a-zA-Z0-9_]+)"
1450 # _setup_teardown() variant has two extra arguments that we ignore
1451 br"(?:\s*,\s*[a-zA-Z0-9_]+\s*,\s*[a-zA-Z0-9_]+)?"
1452 br"\s*\)",
1453 # We don't check how it finishes; we don't care
Anas Nashifaae71d72018-04-21 22:26:48 -05001454 re.MULTILINE)
1455 suite_run_regex = re.compile(
1456 br"^\s*ztest_run_test_suite\((?P<suite_name>[a-zA-Z0-9_]+)\)",
1457 re.MULTILINE)
1458 achtung_regex = re.compile(
1459 br"(#ifdef|#endif)",
1460 re.MULTILINE)
1461 warnings = None
1462
1463 with open(inf_name) as inf:
1464 with contextlib.closing(mmap.mmap(inf.fileno(), 0, mmap.MAP_PRIVATE,
1465 mmap.PROT_READ, 0)) as main_c:
Anas Nashifaae71d72018-04-21 22:26:48 -05001466 suite_regex_match = suite_regex.search(main_c)
1467 if not suite_regex_match:
1468 # can't find ztest_test_suite, maybe a client, because
1469 # it includes ztest.h
1470 return None, None
1471
1472 suite_run_match = suite_run_regex.search(main_c)
1473 if not suite_run_match:
1474 raise ValueError("can't find ztest_run_test_suite")
1475
1476 achtung_matches = re.findall(
1477 achtung_regex,
1478 main_c[suite_regex_match.end():suite_run_match.start()])
1479 if achtung_matches:
1480 warnings = "found invalid %s in ztest_test_suite()" \
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001481 % ", ".join(set([
1482 match.decode() for match in achtung_matches
1483 ]))
1484 _matches = re.findall(
Anas Nashifaae71d72018-04-21 22:26:48 -05001485 stc_regex,
1486 main_c[suite_regex_match.end():suite_run_match.start()])
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001487 matches = [ match.decode().replace("test_", "") for match in _matches ]
Anas Nashifaae71d72018-04-21 22:26:48 -05001488 return matches, warnings
1489
1490 def scan_path(self, path):
1491 subcases = []
1492 for filename in glob.glob(os.path.join(path, "src", "*.c")):
1493 try:
1494 _subcases, warnings = self.scan_file(filename)
1495 if warnings:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001496 error("%s: %s" % (filename, warnings))
Anas Nashifaae71d72018-04-21 22:26:48 -05001497 if _subcases:
1498 subcases += _subcases
1499 except ValueError as e:
1500 error("%s: can't find: %s", filename, e)
1501 return subcases
1502
1503
1504 def parse_subcases(self):
1505 results = self.scan_path(self.code_location)
1506 for sub in results:
Inaky Perez-Gonzalez75e2d902018-04-26 00:17:01 -07001507 name = "{}.{}".format(self.id, sub)
Anas Nashifaae71d72018-04-21 22:26:48 -05001508 self.cases.append(name)
1509
1510
Anas Nashif75547e22018-02-24 08:32:14 -06001511 def __str__(self):
Andrew Boie6acbe632015-07-17 12:03:52 -07001512 return self.name
1513
1514
Andrew Boie6acbe632015-07-17 12:03:52 -07001515class TestInstance:
1516 """Class representing the execution of a particular TestCase on a platform
1517
1518 @param test The TestCase object we want to build/execute
1519 @param platform Platform object that we want to build and run against
1520 @param base_outdir Base directory for all test results. The actual
1521 out directory used is <outdir>/<platform>/<test case name>
1522 """
Anas Nashif3ba1d432017-12-05 15:28:44 -05001523
Anas Nashif37f9dc52018-02-23 08:53:46 -06001524 def __init__(self, test, platform, base_outdir):
Andrew Boie6acbe632015-07-17 12:03:52 -07001525 self.test = test
1526 self.platform = platform
Anas Nashifbd166f42017-09-02 12:32:08 -04001527 self.name = os.path.join(platform.name, test.name)
Andrew Boie6acbe632015-07-17 12:03:52 -07001528 self.outdir = os.path.join(base_outdir, platform.name, test.path)
Anas Nashif37f9dc52018-02-23 08:53:46 -06001529 self.build_only = options.build_only or test.build_only or (test.harness and test.harness != 'console')
Anas Nashife0a6a0b2018-02-15 20:07:24 -06001530 self.results = {}
Andrew Boie6acbe632015-07-17 12:03:52 -07001531
Anas Nashiffa695d22017-10-04 16:14:27 -04001532 def create_overlay(self):
1533 if len(self.test.extra_configs) > 0:
1534 file = os.path.join(self.outdir, "overlay.conf")
1535 os.makedirs(self.outdir, exist_ok=True)
1536 f = open(file, "w")
1537 content = ""
Anas Nashif981f77f2017-10-18 07:53:58 -04001538 content = "\n".join(self.test.extra_configs)
Anas Nashiffa695d22017-10-04 16:14:27 -04001539 f.write(content)
1540 f.close()
1541
Andrew Boie6acbe632015-07-17 12:03:52 -07001542 def calculate_sizes(self):
1543 """Get the RAM/ROM sizes of a test case.
1544
1545 This can only be run after the instance has been executed by
1546 MakeGenerator, otherwise there won't be any binaries to measure.
1547
1548 @return A SizeCalculator object
1549 """
Anas Nashiffb91ad62017-10-31 08:33:17 -04001550 fns = glob.glob(os.path.join(self.outdir, "zephyr", "*.elf"))
Anas Nashif2f4e1702017-11-24 08:11:25 -05001551 fns.extend(glob.glob(os.path.join(self.outdir, "zephyr", "*.exe")))
Anas Nashiff8dcad42016-10-27 18:10:08 -04001552 fns = [x for x in fns if not x.endswith('_prebuilt.elf')]
Andrew Boie5d4eb782015-10-02 10:04:56 -07001553 if (len(fns) != 1):
1554 raise BuildError("Missing/multiple output ELF binary")
Andrew Boie52fef672016-11-29 12:21:59 -08001555 return SizeCalculator(fns[0], self.test.extra_sections)
Andrew Boie6acbe632015-07-17 12:03:52 -07001556
1557 def __repr__(self):
1558 return "<TestCase %s on %s>" % (self.test.name, self.platform.name)
1559
1560
Andrew Boie4ef16c52015-08-28 12:36:03 -07001561def defconfig_cb(context, goals, goal):
1562 if not goal.failed:
1563 return
1564
1565 info("%sCould not build defconfig for %s%s" %
Anas Nashif3ba1d432017-12-05 15:28:44 -05001566 (COLOR_RED, goal.name, COLOR_NORMAL))
Andrew Boie4ef16c52015-08-28 12:36:03 -07001567 if INLINE_LOGS:
1568 with open(goal.get_error_log()) as fp:
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -08001569 data = fp.read()
1570 sys.stdout.write(data)
1571 if log_file:
1572 log_file.write(data)
Andrew Boie4ef16c52015-08-28 12:36:03 -07001573 else:
Andrew Boie08ce5a52016-02-22 13:28:10 -08001574 info("\tsee: " + COLOR_YELLOW + goal.get_error_log() + COLOR_NORMAL)
Andrew Boie4ef16c52015-08-28 12:36:03 -07001575
Andrew Boie6acbe632015-07-17 12:03:52 -07001576
1577class TestSuite:
Andrew Boie60823c22017-02-10 09:43:07 -08001578 config_re = re.compile('(CONFIG_[A-Za-z0-9_]+)[=]\"?([^\"]*)\"?$')
Andrew Boie6acbe632015-07-17 12:03:52 -07001579
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001580 yaml_tc_schema = scl.yaml_load(
Oleg Zhurakivskyy42822082018-08-17 14:54:41 +03001581 os.path.join(ZEPHYR_BASE,
Anas Nashifdb3d55f2017-09-02 06:31:25 -04001582 "scripts", "sanity_chk", "sanitycheck-tc-schema.yaml"))
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001583
Anas Nashif37f9dc52018-02-23 08:53:46 -06001584 def __init__(self, board_root_list, testcase_roots, outdir):
Andrew Boie6acbe632015-07-17 12:03:52 -07001585 # Keep track of which test cases we've filtered out and why
Andrew Boie6acbe632015-07-17 12:03:52 -07001586 self.arches = {}
1587 self.testcases = {}
1588 self.platforms = []
Andrew Boieb391e662015-08-31 15:25:45 -07001589 self.outdir = os.path.abspath(outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07001590 self.instances = {}
1591 self.goals = None
1592 self.discards = None
Kumar Galac84235e2018-04-10 13:32:51 -05001593 self.load_errors = 0
Andrew Boie6acbe632015-07-17 12:03:52 -07001594
Andrew Boie3d348712016-04-08 11:52:13 -07001595 for testcase_root in testcase_roots:
1596 testcase_root = os.path.abspath(testcase_root)
Andrew Boie6acbe632015-07-17 12:03:52 -07001597
Andrew Boie3d348712016-04-08 11:52:13 -07001598 debug("Reading test case configuration files under %s..." %
1599 testcase_root)
1600 for dirpath, dirnames, filenames in os.walk(testcase_root,
1601 topdown=True):
1602 verbose("scanning %s" % dirpath)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001603 if 'sample.yaml' in filenames:
1604 filename = 'sample.yaml'
1605 elif 'testcase.yaml' in filenames:
1606 filename = 'testcase.yaml'
1607 else:
1608 continue
Anas Nashif61e21632018-04-08 13:30:16 -05001609
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001610 verbose("Found possible test case in " + dirpath)
1611 dirnames[:] = []
1612 yaml_path = os.path.join(dirpath, filename)
1613 try:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001614 parsed_data = SanityConfigParser(
1615 yaml_path, self.yaml_tc_schema)
Andrew Boie3d348712016-04-08 11:52:13 -07001616
Paul Sokolovsky100474d2018-01-03 17:19:43 +02001617 workdir = os.path.relpath(dirpath, testcase_root)
Inaky Perez-Gonzalez662dde62017-07-24 10:24:35 -07001618
Paul Sokolovsky100474d2018-01-03 17:19:43 +02001619 for name in parsed_data.tests.keys():
1620 tc_dict = parsed_data.get_test(name, testcase_valid_keys)
1621 tc = TestCase(testcase_root, workdir, name, tc_dict,
1622 yaml_path)
Anas Nashifaae71d72018-04-21 22:26:48 -05001623 tc.parse_subcases()
Anas Nashif7fae29c2017-10-09 13:19:12 -04001624
Paul Sokolovsky100474d2018-01-03 17:19:43 +02001625 self.testcases[tc.name] = tc
1626
1627 except Exception as e:
1628 error("E: %s: can't load (skipping): %s" % (yaml_path, e))
Kumar Galac84235e2018-04-10 13:32:51 -05001629 self.load_errors += 1
Paul Sokolovsky100474d2018-01-03 17:19:43 +02001630
Andrew Boie6acbe632015-07-17 12:03:52 -07001631
Anas Nashif86c8e232017-10-09 13:42:28 -04001632 for board_root in board_root_list:
1633 board_root = os.path.abspath(board_root)
1634
Anas Nashif3ba1d432017-12-05 15:28:44 -05001635 debug(
1636 "Reading platform configuration files under %s..." %
1637 board_root)
Anas Nashif8b11a1f2017-11-26 17:08:47 -05001638 for fn in glob.glob(os.path.join(board_root, "*", "*", "*.yaml")):
1639 verbose("Found plaform configuration " + fn)
1640 try:
1641 platform = Platform(fn)
Anas Nashiff3d48e12018-07-24 08:14:42 -05001642 if platform.sanitycheck:
1643 self.platforms.append(platform)
Anas Nashif8b11a1f2017-11-26 17:08:47 -05001644 except RuntimeError as e:
1645 error("E: %s: can't load: %s" % (fn, e))
Kumar Galac84235e2018-04-10 13:32:51 -05001646 self.load_errors += 1
Andrew Boie6acbe632015-07-17 12:03:52 -07001647
Anas Nashifa792a3d2017-04-04 18:47:49 -04001648 arches = []
1649 for p in self.platforms:
1650 arches.append(p.arch)
1651 for a in list(set(arches)):
Anas Nashif3ba1d432017-12-05 15:28:44 -05001652 aplatforms = [p for p in self.platforms if p.arch == a]
Anas Nashifa792a3d2017-04-04 18:47:49 -04001653 arch = Architecture(a, aplatforms)
1654 self.arches[a] = arch
1655
Andrew Boie6acbe632015-07-17 12:03:52 -07001656 self.instances = {}
1657
1658 def get_last_failed(self):
Anas Nashifb4bdd662018-08-15 17:12:28 -05001659
1660 try:
1661 if not os.path.exists(LAST_SANITY):
1662 raise SanityRuntimeError("Couldn't find last sanity run.")
1663 except Exception as e:
1664 print(str(e))
1665 sys.exit(2)
1666
Andrew Boie6acbe632015-07-17 12:03:52 -07001667 result = []
1668 with open(LAST_SANITY, "r") as fp:
1669 cr = csv.DictReader(fp)
1670 for row in cr:
1671 if row["passed"] == "True":
1672 continue
1673 test = row["test"]
1674 platform = row["platform"]
1675 result.append((test, platform))
1676 return result
1677
Anas Nashifbd166f42017-09-02 12:32:08 -04001678 def load_from_file(self, file):
Anas Nashifb4bdd662018-08-15 17:12:28 -05001679 try:
1680 if not os.path.exists(file):
1681 raise SanityRuntimeError(
1682 "Couldn't find input file with list of tests.")
1683 except Exception as e:
1684 print(str(e))
1685 sys.exit(2)
1686
Anas Nashifbd166f42017-09-02 12:32:08 -04001687 with open(file, "r") as fp:
1688 cr = csv.reader(fp)
1689 instance_list = []
1690 for row in cr:
1691 name = os.path.join(row[0], row[1])
1692 platforms = self.arches[row[3]].platforms
1693 myp = None
1694 for p in platforms:
1695 if p.name == row[2]:
1696 myp = p
1697 break
1698 instance = TestInstance(self.testcases[name], myp, self.outdir)
Anas Nashiffa695d22017-10-04 16:14:27 -04001699 instance.create_overlay()
Anas Nashifbd166f42017-09-02 12:32:08 -04001700 instance_list.append(instance)
1701 self.add_instances(instance_list)
1702
Anas Nashif4f028882017-12-30 11:48:43 -05001703 def apply_filters(self):
Anas Nashifbd166f42017-09-02 12:32:08 -04001704
Anas Nashif7fe35cf2018-02-15 07:20:18 -06001705 toolchain = os.environ.get("ZEPHYR_TOOLCHAIN_VARIANT", None) or \
1706 os.environ.get("ZEPHYR_GCC_VARIANT", None)
Anas Nashifb4bdd662018-08-15 17:12:28 -05001707
1708
1709 try:
1710 if not toolchain:
1711 raise SanityRuntimeError("E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined")
1712 except Exception as e:
1713 print(str(e))
1714 sys.exit(2)
Anas Nashif7fe35cf2018-02-15 07:20:18 -06001715
1716
Andrew Boie6acbe632015-07-17 12:03:52 -07001717 instances = []
1718 discards = {}
Anas Nashif4f028882017-12-30 11:48:43 -05001719 platform_filter = options.platform
1720 last_failed = options.only_failed
1721 testcase_filter = options.test
1722 arch_filter = options.arch
1723 tag_filter = options.tag
1724 exclude_tag = options.exclude_tag
1725 config_filter = options.config
1726 extra_args = options.extra_args
1727 all_plats = options.all
Anas Nashiffa695d22017-10-04 16:14:27 -04001728
Andrew Boie6acbe632015-07-17 12:03:52 -07001729 verbose("platform filter: " + str(platform_filter))
1730 verbose(" arch_filter: " + str(arch_filter))
1731 verbose(" tag_filter: " + str(tag_filter))
Anas Nashifdfa86e22016-10-24 17:08:56 -04001732 verbose(" exclude_tag: " + str(exclude_tag))
Andrew Boie6acbe632015-07-17 12:03:52 -07001733 verbose(" config_filter: " + str(config_filter))
1734
1735 if last_failed:
1736 failed_tests = self.get_last_failed()
1737
Andrew Boie821d8322016-03-22 10:08:35 -07001738 default_platforms = False
1739
1740 if all_plats:
1741 info("Selecting all possible platforms per test case")
1742 # When --all used, any --platform arguments ignored
1743 platform_filter = []
1744 elif not platform_filter:
Andrew Boie6acbe632015-07-17 12:03:52 -07001745 info("Selecting default platforms per test case")
1746 default_platforms = True
Andrew Boie6acbe632015-07-17 12:03:52 -07001747
Sebastian Bøe781e3982017-11-09 11:43:33 +01001748 mg = MakeGenerator(self.outdir)
Anas Nashif2cf0df02015-10-10 09:29:43 -04001749 dlist = {}
Andrew Boie08ce5a52016-02-22 13:28:10 -08001750 for tc_name, tc in self.testcases.items():
1751 for arch_name, arch in self.arches.items():
Anas Nashif2cf0df02015-10-10 09:29:43 -04001752 for plat in arch.platforms:
1753 instance = TestInstance(tc, plat, self.outdir)
1754
Jaakko Hannikainenca505f82016-08-22 15:03:46 +03001755 if (arch_name == "unit") != (tc.type == "unit"):
1756 continue
1757
Anas Nashifbfab06b2017-06-22 09:22:24 -04001758 if tc.build_on_all and not platform_filter:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001759 platform_filter = []
1760
Anas Nashif2bd99bc2015-10-12 13:10:57 -04001761 if tc.skip:
1762 continue
1763
Anas Nashif2cf0df02015-10-10 09:29:43 -04001764 if tag_filter and not tc.tags.intersection(tag_filter):
1765 continue
1766
Anas Nashifdfa86e22016-10-24 17:08:56 -04001767 if exclude_tag and tc.tags.intersection(exclude_tag):
1768 continue
1769
Anas Nashif2cf0df02015-10-10 09:29:43 -04001770 if testcase_filter and tc_name not in testcase_filter:
1771 continue
1772
Anas Nashif3ba1d432017-12-05 15:28:44 -05001773 if last_failed and (
1774 tc.name, plat.name) not in failed_tests:
Anas Nashif2cf0df02015-10-10 09:29:43 -04001775 continue
1776
1777 if arch_filter and arch_name not in arch_filter:
1778 continue
1779
1780 if tc.arch_whitelist and arch.name not in tc.arch_whitelist:
1781 continue
1782
1783 if tc.arch_exclude and arch.name in tc.arch_exclude:
1784 continue
1785
1786 if tc.platform_exclude and plat.name in tc.platform_exclude:
1787 continue
1788
Anas Nashifb17e1ca2017-06-27 18:05:30 -04001789 if tc.toolchain_exclude and toolchain in tc.toolchain_exclude:
1790 continue
1791
Anas Nashif2cf0df02015-10-10 09:29:43 -04001792 if platform_filter and plat.name not in platform_filter:
1793 continue
1794
Anas Nashif62224182017-08-09 23:55:53 -04001795 if plat.ram < tc.min_ram:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001796 continue
1797
1798 if set(plat.ignore_tags) & tc.tags:
1799 continue
1800
Kumar Gala5141d522017-07-07 08:05:48 -05001801 if tc.depends_on:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001802 dep_intersection = tc.depends_on.intersection(
1803 set(plat.supported))
Kumar Gala5141d522017-07-07 08:05:48 -05001804 if dep_intersection != set(tc.depends_on):
1805 continue
Anas Nashifa792a3d2017-04-04 18:47:49 -04001806
1807 if plat.flash < tc.min_flash:
1808 continue
1809
Anas Nashif2cf0df02015-10-10 09:29:43 -04001810 if tc.platform_whitelist and plat.name not in tc.platform_whitelist:
1811 continue
1812
Anas Nashifb17e1ca2017-06-27 18:05:30 -04001813 if tc.toolchain_whitelist and toolchain not in tc.toolchain_whitelist:
1814 continue
1815
Anas Nashif3ba1d432017-12-05 15:28:44 -05001816 if (tc.tc_filter and (plat.default or all_plats or platform_filter)
1817 and toolchain in plat.supported_toolchains):
Anas Nashif8ea9d022015-11-10 12:24:20 -05001818 args = tc.extra_args[:]
Anas Nashiffb91ad62017-10-31 08:33:17 -04001819 args.append("BOARD={}".format(plat.name))
Andrew Boieba612002016-09-01 10:41:03 -07001820 args.extend(extra_args)
Anas Nashif2cf0df02015-10-10 09:29:43 -04001821 # FIXME would be nice to use a common outdir for this so that
Andrew Boie41878222016-11-03 11:58:53 -07001822 # conf, gen_idt, etc aren't rebuilt for every combination,
David B. Kinder29963c32017-06-16 12:32:42 -07001823 # need a way to avoid different Make processes from clobbering
Anas Nashif3ba1d432017-12-05 15:28:44 -05001824 # each other since they all try to build them
1825 # simultaneously
Anas Nashif2cf0df02015-10-10 09:29:43 -04001826
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02001827 if plat.type == "native" and options.enable_coverage:
1828 args.append("CONFIG_COVERAGE=y")
1829
Anas Nashif2cf0df02015-10-10 09:29:43 -04001830 o = os.path.join(self.outdir, plat.name, tc.path)
Anas Nashif13773752018-07-06 18:20:23 -05001831 dlist[tc, plat, tc.name.split("/")[-1]] = os.path.join(o, "zephyr", ".config")
1832 goal = "_".join([plat.name, "_".join(tc.name.split("/")), "config-sanitycheck"])
1833 mg.add_build_goal(goal, os.path.join(ZEPHYR_BASE, tc.code_location),
1834 o, args, "config-sanitycheck.log", make_args="config-sanitycheck")
Anas Nashif2cf0df02015-10-10 09:29:43 -04001835
1836 info("Building testcase defconfigs...")
1837 results = mg.execute(defconfig_cb)
1838
Andrew Boie08ce5a52016-02-22 13:28:10 -08001839 for name, goal in results.items():
Anas Nashifb4bdd662018-08-15 17:12:28 -05001840 try:
1841 if goal.failed:
1842 raise SanityRuntimeError("Couldn't build some defconfigs")
1843 except Exception as e:
1844 error(str(e))
1845 sys.exit(2)
1846
Anas Nashif2cf0df02015-10-10 09:29:43 -04001847
Andrew Boie08ce5a52016-02-22 13:28:10 -08001848 for k, out_config in dlist.items():
Andrew Boie41878222016-11-03 11:58:53 -07001849 test, plat, name = k
Anas Nashif2cf0df02015-10-10 09:29:43 -04001850 defconfig = {}
1851 with open(out_config, "r") as fp:
1852 for line in fp.readlines():
1853 m = TestSuite.config_re.match(line)
1854 if not m:
Andrew Boie3ea78922016-03-24 14:46:00 -07001855 if line.strip() and not line.startswith("#"):
1856 sys.stderr.write("Unrecognized line %s\n" % line)
Anas Nashif2cf0df02015-10-10 09:29:43 -04001857 continue
1858 defconfig[m.group(1)] = m.group(2).strip()
Andrew Boie41878222016-11-03 11:58:53 -07001859 test.defconfig[plat] = defconfig
Anas Nashif2cf0df02015-10-10 09:29:43 -04001860
Andrew Boie08ce5a52016-02-22 13:28:10 -08001861 for tc_name, tc in self.testcases.items():
1862 for arch_name, arch in self.arches.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07001863 instance_list = []
1864 for plat in arch.platforms:
1865 instance = TestInstance(tc, plat, self.outdir)
1866
Jaakko Hannikainenca505f82016-08-22 15:03:46 +03001867 if (arch_name == "unit") != (tc.type == "unit"):
1868 # Discard silently
1869 continue
1870
Anas Nashif2bd99bc2015-10-12 13:10:57 -04001871 if tc.skip:
1872 discards[instance] = "Skip filter"
1873 continue
1874
Anas Nashifbfab06b2017-06-22 09:22:24 -04001875 if tc.build_on_all and not platform_filter:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001876 platform_filter = []
1877
Andrew Boie6acbe632015-07-17 12:03:52 -07001878 if tag_filter and not tc.tags.intersection(tag_filter):
1879 discards[instance] = "Command line testcase tag filter"
1880 continue
1881
Anas Nashifdfa86e22016-10-24 17:08:56 -04001882 if exclude_tag and tc.tags.intersection(exclude_tag):
1883 discards[instance] = "Command line testcase exclude filter"
1884 continue
1885
Andrew Boie6acbe632015-07-17 12:03:52 -07001886 if testcase_filter and tc_name not in testcase_filter:
1887 discards[instance] = "Testcase name filter"
1888 continue
1889
Anas Nashif3ba1d432017-12-05 15:28:44 -05001890 if last_failed and (
1891 tc.name, plat.name) not in failed_tests:
Andrew Boie6acbe632015-07-17 12:03:52 -07001892 discards[instance] = "Passed or skipped during last run"
1893 continue
1894
1895 if arch_filter and arch_name not in arch_filter:
1896 discards[instance] = "Command line testcase arch filter"
1897 continue
1898
1899 if tc.arch_whitelist and arch.name not in tc.arch_whitelist:
1900 discards[instance] = "Not in test case arch whitelist"
1901 continue
1902
Anas Nashif30d13872015-10-05 10:02:45 -04001903 if tc.arch_exclude and arch.name in tc.arch_exclude:
1904 discards[instance] = "In test case arch exclude"
1905 continue
1906
1907 if tc.platform_exclude and plat.name in tc.platform_exclude:
1908 discards[instance] = "In test case platform exclude"
1909 continue
1910
Anas Nashifb17e1ca2017-06-27 18:05:30 -04001911 if tc.toolchain_exclude and toolchain in tc.toolchain_exclude:
1912 discards[instance] = "In test case toolchain exclude"
1913 continue
1914
Andrew Boie6acbe632015-07-17 12:03:52 -07001915 if platform_filter and plat.name not in platform_filter:
1916 discards[instance] = "Command line platform filter"
1917 continue
1918
1919 if tc.platform_whitelist and plat.name not in tc.platform_whitelist:
1920 discards[instance] = "Not in testcase platform whitelist"
1921 continue
1922
Anas Nashifb17e1ca2017-06-27 18:05:30 -04001923 if tc.toolchain_whitelist and toolchain not in tc.toolchain_whitelist:
1924 discards[instance] = "Not in testcase toolchain whitelist"
1925 continue
1926
Anas Nashif86c8e232017-10-09 13:42:28 -04001927 if toolchain and toolchain not in plat.supported_toolchains and tc.type != 'unit':
Javier B Perez4b554ba2016-08-15 13:25:33 -05001928 discards[instance] = "Not supported by the toolchain"
1929 continue
1930
Anas Nashif62224182017-08-09 23:55:53 -04001931 if plat.ram < tc.min_ram:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001932 discards[instance] = "Not enough RAM"
1933 continue
1934
Kumar Gala5141d522017-07-07 08:05:48 -05001935 if tc.depends_on:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001936 dep_intersection = tc.depends_on.intersection(
1937 set(plat.supported))
Kumar Gala5141d522017-07-07 08:05:48 -05001938 if dep_intersection != set(tc.depends_on):
1939 discards[instance] = "No hardware support"
1940 continue
Anas Nashifa792a3d2017-04-04 18:47:49 -04001941
Anas Nashif62224182017-08-09 23:55:53 -04001942 if plat.flash < tc.min_flash:
Anas Nashifa792a3d2017-04-04 18:47:49 -04001943 discards[instance] = "Not enough FLASH"
1944 continue
1945
1946 if set(plat.ignore_tags) & tc.tags:
1947 discards[instance] = "Excluded tags per platform"
1948 continue
1949
Anas Nashif674bb282018-01-09 09:12:15 -05001950 defconfig = {
Anas Nashif674bb282018-01-09 09:12:15 -05001951 "ARCH": arch.name,
1952 "PLATFORM": plat.name
1953 }
Javier B Perez79414542016-08-08 12:24:59 -05001954 defconfig.update(os.environ)
Andrew Boie41878222016-11-03 11:58:53 -07001955 for p, tdefconfig in tc.defconfig.items():
1956 if p == plat:
Andrew Boie3ea78922016-03-24 14:46:00 -07001957 defconfig.update(tdefconfig)
Anas Nashif2cf0df02015-10-10 09:29:43 -04001958 break
1959
Andrew Boie3ea78922016-03-24 14:46:00 -07001960 if tc.tc_filter:
1961 try:
1962 res = expr_parser.parse(tc.tc_filter, defconfig)
Andrew Boiec09b4b82017-04-18 11:46:07 -07001963 except (ValueError, SyntaxError) as se:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001964 sys.stderr.write(
1965 "Failed processing %s\n" % tc.yamlfile)
Andrew Boie3ea78922016-03-24 14:46:00 -07001966 raise se
1967 if not res:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001968 discards[instance] = (
1969 "defconfig doesn't satisfy expression '%s'" %
1970 tc.tc_filter)
Andrew Boie3ea78922016-03-24 14:46:00 -07001971 continue
Anas Nashif2cf0df02015-10-10 09:29:43 -04001972
Andrew Boie6acbe632015-07-17 12:03:52 -07001973 instance_list.append(instance)
1974
1975 if not instance_list:
1976 # Every platform in this arch was rejected already
1977 continue
1978
Anas Nashifa792a3d2017-04-04 18:47:49 -04001979 if default_platforms and not tc.build_on_all:
1980 if not tc.platform_whitelist:
Anas Nashif3ba1d432017-12-05 15:28:44 -05001981 instances = list(
1982 filter(
1983 lambda tc: tc.platform.default,
1984 instance_list))
Anas Nashifa792a3d2017-04-04 18:47:49 -04001985 self.add_instances(instances)
1986 else:
Anas Nashifab747062017-12-05 17:59:01 -05001987 self.add_instances(instance_list[:1])
Anas Nashifa792a3d2017-04-04 18:47:49 -04001988
Anas Nashif3ba1d432017-12-05 15:28:44 -05001989 for instance in list(
1990 filter(lambda tc: not tc.platform.default, instance_list)):
Anas Nashifa792a3d2017-04-04 18:47:49 -04001991 discards[instance] = "Not a default test platform"
Andrew Boie6acbe632015-07-17 12:03:52 -07001992 else:
Andrew Boie821d8322016-03-22 10:08:35 -07001993 self.add_instances(instance_list)
Anas Nashifab351f42018-04-08 08:57:48 -05001994
1995 for name, case in self.instances.items():
1996 case.create_overlay()
1997
Andrew Boie6acbe632015-07-17 12:03:52 -07001998 self.discards = discards
1999 return discards
2000
Andrew Boie821d8322016-03-22 10:08:35 -07002001 def add_instances(self, ti_list):
2002 for ti in ti_list:
2003 self.instances[ti.name] = ti
Andrew Boie6acbe632015-07-17 12:03:52 -07002004
Anas Nashif37f9dc52018-02-23 08:53:46 -06002005 def execute(self, cb, cb_context):
Daniel Leung6b170072016-04-07 12:10:25 -07002006
2007 def calc_one_elf_size(name, goal):
2008 if not goal.failed:
Alberto Escolar Piedrasb1045fe2018-07-14 13:11:02 +02002009 if self.instances[name].platform.type != "native":
2010 i = self.instances[name]
2011 sc = i.calculate_sizes()
2012 goal.metrics["ram_size"] = sc.get_ram_size()
2013 goal.metrics["rom_size"] = sc.get_rom_size()
2014 goal.metrics["unrecognized"] = sc.unrecognized_sections()
2015 else:
2016 goal.metrics["ram_size"] = 0
2017 goal.metrics["rom_size"] = 0
2018 goal.metrics["unrecognized"] = []
Daniel Leung6b170072016-04-07 12:10:25 -07002019
Anas Nashif37f9dc52018-02-23 08:53:46 -06002020 mg = MakeGenerator(self.outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07002021 for i in self.instances.values():
Anas Nashif37f9dc52018-02-23 08:53:46 -06002022 mg.add_test_instance(i, options.extra_args)
Andrew Boie6acbe632015-07-17 12:03:52 -07002023 self.goals = mg.execute(cb, cb_context)
Daniel Leung6b170072016-04-07 12:10:25 -07002024
2025 # Parallelize size calculation
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03002026 executor = concurrent.futures.ThreadPoolExecutor(JOBS)
Anas Nashif3ba1d432017-12-05 15:28:44 -05002027 futures = [executor.submit(calc_one_elf_size, name, goal)
2028 for name, goal in self.goals.items()]
Daniel Leung6b170072016-04-07 12:10:25 -07002029 concurrent.futures.wait(futures)
2030
Andrew Boie6acbe632015-07-17 12:03:52 -07002031 return self.goals
2032
Anas Nashifbd166f42017-09-02 12:32:08 -04002033 def run_report(self, filename):
2034 with open(filename, "at") as csvfile:
2035 fieldnames = ['path', 'test', 'platform', 'arch']
2036 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2037 for instance in self.instances.values():
2038 rowdict = {
Anas Nashif3ba1d432017-12-05 15:28:44 -05002039 "path": os.path.dirname(instance.test.name),
2040 "test": os.path.basename(instance.test.name),
2041 "platform": instance.platform.name,
2042 "arch": instance.platform.arch
2043 }
Anas Nashifbd166f42017-09-02 12:32:08 -04002044 cw.writerow(rowdict)
2045
Andrew Boie6acbe632015-07-17 12:03:52 -07002046 def discard_report(self, filename):
Anas Nashifb4bdd662018-08-15 17:12:28 -05002047
2048 try:
2049 if self.discards is None:
2050 raise SanityRuntimeError("apply_filters() hasn't been run!")
2051 except Exception as e:
2052 error(str(e))
2053 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -07002054
Anas Nashifbd166f42017-09-02 12:32:08 -04002055 with open(filename, "wt") as csvfile:
Andrew Boie6acbe632015-07-17 12:03:52 -07002056 fieldnames = ["test", "arch", "platform", "reason"]
2057 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2058 cw.writeheader()
Andrew Boie08ce5a52016-02-22 13:28:10 -08002059 for instance, reason in self.discards.items():
Anas Nashif3ba1d432017-12-05 15:28:44 -05002060 rowdict = {"test": instance.test.name,
2061 "arch": instance.platform.arch,
2062 "platform": instance.platform.name,
2063 "reason": reason}
Andrew Boie6acbe632015-07-17 12:03:52 -07002064 cw.writerow(rowdict)
2065
2066 def compare_metrics(self, filename):
2067 # name, datatype, lower results better
2068 interesting_metrics = [("ram_size", int, True),
2069 ("rom_size", int, True)]
2070
Anas Nashifb4bdd662018-08-15 17:12:28 -05002071 try:
2072 if self.goals is None:
2073 raise SanityRuntimeError("execute() hasn't been run!")
2074 except Exception as e:
2075 print(str(e))
2076 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -07002077
2078 if not os.path.exists(filename):
2079 info("Cannot compare metrics, %s not found" % filename)
2080 return []
2081
2082 results = []
2083 saved_metrics = {}
2084 with open(filename) as fp:
2085 cr = csv.DictReader(fp)
2086 for row in cr:
2087 d = {}
2088 for m, _, _ in interesting_metrics:
2089 d[m] = row[m]
2090 saved_metrics[(row["test"], row["platform"])] = d
2091
Andrew Boie08ce5a52016-02-22 13:28:10 -08002092 for name, goal in self.goals.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07002093 i = self.instances[name]
2094 mkey = (i.test.name, i.platform.name)
2095 if mkey not in saved_metrics:
2096 continue
2097 sm = saved_metrics[mkey]
2098 for metric, mtype, lower_better in interesting_metrics:
2099 if metric not in goal.metrics:
2100 continue
2101 if sm[metric] == "":
2102 continue
2103 delta = goal.metrics[metric] - mtype(sm[metric])
Andrew Boieea7928f2015-08-14 14:27:38 -07002104 if delta == 0:
2105 continue
2106 results.append((i, metric, goal.metrics[metric], delta,
2107 lower_better))
Andrew Boie6acbe632015-07-17 12:03:52 -07002108 return results
2109
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002110
2111
2112 def encode_for_xml(self, unicode_data, encoding='ascii'):
2113 unicode_data = unicode_data.replace('\x00', '')
2114 return unicode_data
2115
2116 def testcase_target_report(self, report_file):
2117
2118 run = "Sanitycheck"
2119 eleTestsuite = None
2120 append = options.only_failed
2121
2122 errors = 0
2123 passes = 0
2124 fails = 0
2125 duration = 0
2126 skips = 0
2127
2128 for identifier, ti in self.instances.items():
2129 for k in ti.results.keys():
2130 if ti.results[k] == 'PASS':
2131 passes += 1
2132 elif ti.results[k] == 'BLOCK':
2133 errors += 1
Anas Nashif61e21632018-04-08 13:30:16 -05002134 elif ti.results[k] == 'SKIP':
2135 skips += 1
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002136 else:
2137 fails += 1
2138
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002139 eleTestsuites = ET.Element('testsuites')
2140 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
2141 name=run, time="%d" % duration,
2142 tests="%d" % (errors + passes + fails),
2143 failures="%d" % fails,
Anas Nashif61e21632018-04-08 13:30:16 -05002144 errors="%d" % errors, skipped="%d" %skips)
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002145
2146 handler_time = "0"
Anas Nashif61e21632018-04-08 13:30:16 -05002147
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002148 # print out test results
2149 for identifier, ti in self.instances.items():
2150 for k in ti.results.keys():
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002151
2152 eleTestcase = ET.SubElement(
2153 eleTestsuite, 'testcase', classname="%s:%s" %(ti.platform.name, os.path.basename(ti.test.name)),
Anas Nashif61e21632018-04-08 13:30:16 -05002154 name="%s" % (k), time=handler_time)
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002155 if ti.results[k] in ['FAIL', 'BLOCK']:
2156 el = None
2157
2158 if ti.results[k] == 'FAIL':
2159 el = ET.SubElement(
2160 eleTestcase,
2161 'failure',
2162 type="failure",
2163 message="failed")
2164 elif ti.results[k] == 'BLOCK':
2165 el = ET.SubElement(
2166 eleTestcase,
2167 'error',
2168 type="failure",
2169 message="failed")
2170 p = os.path.join(options.outdir, ti.platform.name, ti.test.name)
2171 bl = os.path.join(p, "handler.log")
2172
2173 if os.path.exists(bl):
2174 with open(bl, "rb") as f:
2175 log = f.read().decode("utf-8")
2176 el.text = self.encode_for_xml(log)
2177
Anas Nashif61e21632018-04-08 13:30:16 -05002178 elif ti.results[k] == 'SKIP':
2179 el = ET.SubElement(
2180 eleTestcase,
2181 'skipped')
2182
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002183 result = ET.tostring(eleTestsuites)
2184 f = open(report_file, 'wb')
2185 f.write(result)
2186 f.close()
2187
2188
Anas Nashif4f028882017-12-30 11:48:43 -05002189 def testcase_xunit_report(self, filename, duration):
Anas Nashifb4bdd662018-08-15 17:12:28 -05002190 try:
2191 if self.goals is None:
2192 raise SanityRuntimeError("execute() hasn't been run!")
2193 except Exception as e:
2194 print(str(e))
2195 sys.exit(2)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002196
2197 fails = 0
2198 passes = 0
2199 errors = 0
2200
2201 for name, goal in self.goals.items():
2202 if goal.failed:
Anas Nashif9a839df2018-01-29 08:42:38 -05002203 if goal.reason in ['build_error', 'handler_crash']:
Anas Nashifb3311ed2017-04-13 14:44:48 -04002204 errors += 1
2205 else:
2206 fails += 1
2207 else:
2208 passes += 1
2209
2210 run = "Sanitycheck"
2211 eleTestsuite = None
Anas Nashif4f028882017-12-30 11:48:43 -05002212 append = options.only_failed
Anas Nashifb3311ed2017-04-13 14:44:48 -04002213
Anas Nashif0605fa32017-05-07 08:51:02 -04002214 if os.path.exists(filename) and append:
Anas Nashifb3311ed2017-04-13 14:44:48 -04002215 tree = ET.parse(filename)
2216 eleTestsuites = tree.getroot()
Anas Nashif3ba1d432017-12-05 15:28:44 -05002217 eleTestsuite = tree.findall('testsuite')[0]
Anas Nashifb3311ed2017-04-13 14:44:48 -04002218 else:
2219 eleTestsuites = ET.Element('testsuites')
Anas Nashif3ba1d432017-12-05 15:28:44 -05002220 eleTestsuite = ET.SubElement(eleTestsuites, 'testsuite',
2221 name=run, time="%d" % duration,
2222 tests="%d" % (errors + passes + fails),
2223 failures="%d" % fails,
2224 errors="%d" % errors, skip="0")
Anas Nashifb3311ed2017-04-13 14:44:48 -04002225
Anas Nashifc8390f12017-11-25 17:14:12 -05002226 handler_time = "0"
Anas Nashifb3311ed2017-04-13 14:44:48 -04002227 for name, goal in self.goals.items():
2228
2229 i = self.instances[name]
2230 if append:
2231 for tc in eleTestsuite.findall('testcase'):
Anas Nashif3ba1d432017-12-05 15:28:44 -05002232 if tc.get('classname') == "%s:%s" % (
2233 i.platform.name, i.test.name):
Anas Nashifb3311ed2017-04-13 14:44:48 -04002234 eleTestsuite.remove(tc)
2235
Anas Nashif4d25b502017-11-25 17:37:17 -05002236 if not goal.failed and goal.handler:
2237 handler_time = "%s" %(goal.metrics["handler_time"])
Anas Nashifb3311ed2017-04-13 14:44:48 -04002238
Anas Nashif3ba1d432017-12-05 15:28:44 -05002239 eleTestcase = ET.SubElement(
2240 eleTestsuite, 'testcase', classname="%s:%s" %
2241 (i.platform.name, i.test.name), name="%s" %
2242 (name), time=handler_time)
Anas Nashifb3311ed2017-04-13 14:44:48 -04002243 if goal.failed:
Anas Nashif3ba1d432017-12-05 15:28:44 -05002244 failure = ET.SubElement(
2245 eleTestcase,
2246 'failure',
2247 type="failure",
2248 message=goal.reason)
Anas Nashif4f028882017-12-30 11:48:43 -05002249 p = ("%s/%s/%s" % (options.outdir, i.platform.name, i.test.name))
Anas Nashifb3311ed2017-04-13 14:44:48 -04002250 bl = os.path.join(p, "build.log")
Anas Nashifaccc8eb2017-05-01 16:33:43 -04002251 if goal.reason != 'build_error':
Anas Nashifa49048b2018-01-29 08:41:19 -05002252 bl = os.path.join(p, "handler.log")
Anas Nashifaccc8eb2017-05-01 16:33:43 -04002253
Anas Nashifb3311ed2017-04-13 14:44:48 -04002254 if os.path.exists(bl):
Anas Nashif712d3452017-12-29 22:09:03 -05002255 with open(bl, "rb") as f:
2256 log = f.read().decode("utf-8")
2257 failure.text = log
Anas Nashifb3311ed2017-04-13 14:44:48 -04002258
2259 result = ET.tostring(eleTestsuites)
2260 f = open(filename, 'wb')
2261 f.write(result)
2262 f.close()
2263
Andrew Boie6acbe632015-07-17 12:03:52 -07002264 def testcase_report(self, filename):
Anas Nashifb4bdd662018-08-15 17:12:28 -05002265 try:
2266 if self.goals is None:
2267 raise SanityRuntimeError("execute() hasn't been run!")
2268 except Exception as e:
2269 print(str(e))
2270 sys.exit(2)
Andrew Boie6acbe632015-07-17 12:03:52 -07002271
Andrew Boie08ce5a52016-02-22 13:28:10 -08002272 with open(filename, "wt") as csvfile:
Andrew Boie6acbe632015-07-17 12:03:52 -07002273 fieldnames = ["test", "arch", "platform", "passed", "status",
Anas Nashifc8390f12017-11-25 17:14:12 -05002274 "extra_args", "qemu", "handler_time", "ram_size",
Andrew Boie6acbe632015-07-17 12:03:52 -07002275 "rom_size"]
2276 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2277 cw.writeheader()
Andrew Boie08ce5a52016-02-22 13:28:10 -08002278 for name, goal in self.goals.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07002279 i = self.instances[name]
Anas Nashif3ba1d432017-12-05 15:28:44 -05002280 rowdict = {"test": i.test.name,
2281 "arch": i.platform.arch,
2282 "platform": i.platform.name,
2283 "extra_args": " ".join(i.test.extra_args),
2284 "qemu": i.platform.qemu_support}
Andrew Boie6acbe632015-07-17 12:03:52 -07002285 if goal.failed:
2286 rowdict["passed"] = False
2287 rowdict["status"] = goal.reason
2288 else:
2289 rowdict["passed"] = True
Anas Nashif4d25b502017-11-25 17:37:17 -05002290 if goal.handler:
Anas Nashifc8390f12017-11-25 17:14:12 -05002291 rowdict["handler_time"] = goal.metrics["handler_time"]
Andrew Boie6acbe632015-07-17 12:03:52 -07002292 rowdict["ram_size"] = goal.metrics["ram_size"]
2293 rowdict["rom_size"] = goal.metrics["rom_size"]
2294 cw.writerow(rowdict)
2295
2296
2297def parse_arguments():
2298
Anas Nashif3ba1d432017-12-05 15:28:44 -05002299 parser = argparse.ArgumentParser(
2300 description=__doc__,
2301 formatter_class=argparse.RawDescriptionHelpFormatter)
Genaro Saucedo Tejada28bba922016-10-24 18:00:58 -05002302 parser.fromfile_prefix_chars = "+"
Andrew Boie6acbe632015-07-17 12:03:52 -07002303
Anas Nashif3ba1d432017-12-05 15:28:44 -05002304 parser.add_argument(
2305 "-p", "--platform", action="append",
2306 help="Platform filter for testing. This option may be used multiple "
2307 "times. Testcases will only be built/run on the platforms "
2308 "specified. If this option is not used, then platforms marked "
2309 "as default in the platform metadata file will be chosen "
2310 "to build and test. ")
2311 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05002312 "-a", "--arch", action="append",
2313 help="Arch filter for testing. Takes precedence over --platform. "
2314 "If unspecified, test all arches. Multiple invocations "
2315 "are treated as a logical 'or' relationship")
2316 parser.add_argument(
2317 "-t", "--tag", action="append",
2318 help="Specify tags to restrict which tests to run by tag value. "
2319 "Default is to not do any tag filtering. Multiple invocations "
2320 "are treated as a logical 'or' relationship")
Anas Nashifdfa86e22016-10-24 17:08:56 -04002321 parser.add_argument("-e", "--exclude-tag", action="append",
Anas Nashif3ba1d432017-12-05 15:28:44 -05002322 help="Specify tags of tests that should not run. "
2323 "Default is to run all tests with all tags.")
2324 parser.add_argument(
2325 "-f",
2326 "--only-failed",
2327 action="store_true",
2328 help="Run only those tests that failed the previous sanity check "
2329 "invocation.")
2330 parser.add_argument(
2331 "-c", "--config", action="append",
2332 help="Specify platform configuration values filtering. This can be "
2333 "specified two ways: <config>=<value> or just <config>. The "
2334 "defconfig for all platforms will be "
2335 "checked. For the <config>=<value> case, only match defconfig "
2336 "that have that value defined. For the <config> case, match "
2337 "defconfig that have that value assigned to any value. "
2338 "Prepend a '!' to invert the match.")
2339 parser.add_argument(
2340 "-s", "--test", action="append",
2341 help="Run only the specified test cases. These are named by "
2342 "<path to test project relative to "
2343 "--testcase-root>/<testcase.yaml section name>")
2344 parser.add_argument(
2345 "-l", "--all", action="store_true",
2346 help="Build/test on all platforms. Any --platform arguments "
2347 "ignored.")
Andrew Boie6acbe632015-07-17 12:03:52 -07002348
Anas Nashif3ba1d432017-12-05 15:28:44 -05002349 parser.add_argument(
2350 "-o", "--testcase-report",
2351 help="Output a CSV spreadsheet containing results of the test run")
2352 parser.add_argument(
2353 "-d", "--discard-report",
2354 help="Output a CSV spreadsheet showing tests that were skipped "
2355 "and why")
Daniel Leung7f850102016-04-08 11:07:32 -07002356 parser.add_argument("--compare-report",
Anas Nashif3ba1d432017-12-05 15:28:44 -05002357 help="Use this report file for size comparison")
Daniel Leung7f850102016-04-08 11:07:32 -07002358
Anas Nashif3ba1d432017-12-05 15:28:44 -05002359 parser.add_argument(
2360 "-B", "--subset",
2361 help="Only run a subset of the tests, 1/4 for running the first 25%%, "
2362 "3/5 means run the 3rd fifth of the total. "
2363 "This option is useful when running a large number of tests on "
2364 "different hosts to speed up execution time.")
Anas Nashifa8a13882017-12-30 13:01:06 -05002365
2366 parser.add_argument(
2367 "-N", "--ninja", action="store_true",
Anas Nashif25f6ab62018-03-06 07:15:11 -06002368 help="Use the Ninja generator with CMake")
Anas Nashifa8a13882017-12-30 13:01:06 -05002369
Anas Nashif3ba1d432017-12-05 15:28:44 -05002370 parser.add_argument(
2371 "-y", "--dry-run", action="store_true",
2372 help="Create the filtered list of test cases, but don't actually "
2373 "run them. Useful if you're just interested in "
2374 "--discard-report")
Andrew Boie6acbe632015-07-17 12:03:52 -07002375
Anas Nashif75547e22018-02-24 08:32:14 -06002376 parser.add_argument("--list-tags", action="store_true",
2377 help="list all tags in selected tests")
2378
Anas Nashifc0149cc2018-04-14 23:12:58 -05002379 parser.add_argument("--list-tests", action="store_true",
2380 help="list all tests.")
2381
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05002382 parser.add_argument("--export-tests", action="store",
2383 metavar="FILENAME",
2384 help="Export tests case meta-data to a file in CSV format.")
2385
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002386 parser.add_argument("--detailed-report",
2387 action="store",
2388 metavar="FILENAME",
2389 help="Generate a junit report with detailed testcase results.")
2390
Anas Nashif3ba1d432017-12-05 15:28:44 -05002391 parser.add_argument(
2392 "-r", "--release", action="store_true",
2393 help="Update the benchmark database with the results of this test "
2394 "run. Intended to be run by CI when tagging an official "
2395 "release. This database is used as a basis for comparison "
2396 "when looking for deltas in metrics such as footprint")
Andrew Boie6acbe632015-07-17 12:03:52 -07002397 parser.add_argument("-w", "--warnings-as-errors", action="store_true",
Anas Nashif3ba1d432017-12-05 15:28:44 -05002398 help="Treat warning conditions as errors")
2399 parser.add_argument(
2400 "-v",
2401 "--verbose",
2402 action="count",
2403 default=0,
2404 help="Emit debugging information, call multiple times to increase "
2405 "verbosity")
2406 parser.add_argument(
2407 "-i", "--inline-logs", action="store_true",
2408 help="Upon test failure, print relevant log data to stdout "
2409 "instead of just a path to it")
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -08002410 parser.add_argument("--log-file", metavar="FILENAME", action="store",
Anas Nashif3ba1d432017-12-05 15:28:44 -05002411 help="log also to file")
2412 parser.add_argument(
2413 "-m", "--last-metrics", action="store_true",
2414 help="Instead of comparing metrics from the last --release, "
2415 "compare with the results of the previous sanity check "
2416 "invocation")
2417 parser.add_argument(
2418 "-u",
2419 "--no-update",
2420 action="store_true",
2421 help="do not update the results of the last run of the sanity "
2422 "checks")
2423 parser.add_argument(
2424 "-F",
2425 "--load-tests",
2426 metavar="FILENAME",
2427 action="store",
2428 help="Load list of tests to be run from file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04002429
Anas Nashif3ba1d432017-12-05 15:28:44 -05002430 parser.add_argument(
2431 "-E",
2432 "--save-tests",
2433 metavar="FILENAME",
2434 action="store",
2435 help="Save list of tests to be run to file.")
Anas Nashifbd166f42017-09-02 12:32:08 -04002436
Anas Nashif3ba1d432017-12-05 15:28:44 -05002437 parser.add_argument(
2438 "-b", "--build-only", action="store_true",
2439 help="Only build the code, do not execute any of it in QEMU")
2440 parser.add_argument(
2441 "-j", "--jobs", type=int,
Oleg Zhurakivskyyf3bc9672018-08-17 18:31:38 +03002442 help="Number of jobs for building, defaults to number of CPU threads "
2443 "overcommited by factor 2")
Anas Nashif73440ea2018-02-19 10:57:03 -06002444
2445 parser.add_argument(
2446 "--device-testing", action="store_true",
Anas Nashif333a3152018-05-24 14:35:33 -05002447 help="Test on device directly. Specify the serial device to "
2448 "use with the --device-serial option.")
Anas Nashif73440ea2018-02-19 10:57:03 -06002449 parser.add_argument(
2450 "--device-serial",
Anas Nashif333a3152018-05-24 14:35:33 -05002451 help="Serial device for accessing the board (e.g., /dev/ttyACM0)")
Anas Nashif3ba1d432017-12-05 15:28:44 -05002452 parser.add_argument(
Anas Nashif424a3db2018-02-20 08:37:24 -06002453 "--show-footprint", action="store_true",
2454 help="Show footprint statistics and deltas since last release."
2455 )
2456 parser.add_argument(
Anas Nashif3ba1d432017-12-05 15:28:44 -05002457 "-H", "--footprint-threshold", type=float, default=5,
2458 help="When checking test case footprint sizes, warn the user if "
2459 "the new app size is greater then the specified percentage "
2460 "from the last release. Default is 5. 0 to warn on any "
2461 "increase on app size")
2462 parser.add_argument(
2463 "-D", "--all-deltas", action="store_true",
2464 help="Show all footprint deltas, positive or negative. Implies "
2465 "--footprint-threshold=0")
Håkon Øye Amundsende223cc2018-04-25 06:24:25 +00002466 parser.add_argument(
2467 "-O", "--outdir",
2468 default="%s/sanity-out" % ZEPHYR_BASE,
2469 help="Output directory for logs and binaries. "
2470 "This directory will be deleted unless '--no-clean' is set.")
Anas Nashif3ba1d432017-12-05 15:28:44 -05002471 parser.add_argument(
2472 "-n", "--no-clean", action="store_true",
2473 help="Do not delete the outdir before building. Will result in "
2474 "faster compilation since builds will be incremental")
2475 parser.add_argument(
2476 "-T", "--testcase-root", action="append", default=[],
2477 help="Base directory to recursively search for test cases. All "
2478 "testcase.yaml files under here will be processed. May be "
2479 "called multiple times. Defaults to the 'samples' and "
2480 "'tests' directories in the Zephyr tree.")
2481 board_root_list = ["%s/boards" % ZEPHYR_BASE,
2482 "%s/scripts/sanity_chk/boards" % ZEPHYR_BASE]
2483 parser.add_argument(
2484 "-A", "--board-root", action="append", default=board_root_list,
2485 help="Directory to search for board configuration files. All .yaml "
2486 "files in the directory will be processed.")
2487 parser.add_argument(
2488 "-z", "--size", action="append",
2489 help="Don't run sanity checks. Instead, produce a report to "
2490 "stdout detailing RAM/ROM sizes on the specified filenames. "
2491 "All other command line arguments ignored.")
2492 parser.add_argument(
2493 "-S", "--enable-slow", action="store_true",
2494 help="Execute time-consuming test cases that have been marked "
2495 "as 'slow' in testcase.yaml. Normally these are only built.")
Andrew Boie55121052016-07-20 11:52:04 -07002496 parser.add_argument("-R", "--enable-asserts", action="store_true",
Kumar Gala0d48cbe2018-01-26 10:22:20 -06002497 default=True,
Andrew Boie29599f62018-05-24 13:33:09 -07002498 help="deprecated, left for compatibility")
Kumar Gala0d48cbe2018-01-26 10:22:20 -06002499 parser.add_argument("--disable-asserts", action="store_false",
2500 dest="enable_asserts",
Andrew Boie29599f62018-05-24 13:33:09 -07002501 help="deprecated, left for compatibility")
Anas Nashife3febe92016-11-30 14:25:44 -05002502 parser.add_argument("-Q", "--error-on-deprecations", action="store_false",
Anas Nashif3ba1d432017-12-05 15:28:44 -05002503 help="Error on deprecation warnings.")
Sebastian Bøec2182612017-11-09 12:25:02 +01002504
2505 parser.add_argument(
2506 "-x", "--extra-args", action="append", default=[],
Alberto Escolar Piedras25e06362018-02-10 17:40:33 +01002507 help="""Extra CMake cache entries to define when building test cases.
2508 May be called multiple times. The key-value entries will be
Sebastian Bøec2182612017-11-09 12:25:02 +01002509 prefixed with -D before being passed to CMake.
2510
2511 E.g
2512 "sanitycheck -x=USE_CCACHE=0"
2513 will translate to
2514 "cmake -DUSE_CCACHE=0"
2515
2516 which will ultimately disable ccache.
2517 """
2518 )
2519
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02002520 parser.add_argument("--enable-coverage", action="store_true",
2521 help="Enable code coverage when building unit tests and"
2522 " when targeting the native_posix board")
2523
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03002524 parser.add_argument("-C", "--coverage", action="store_true",
Alberto Escolar Piedras25e06362018-02-10 17:40:33 +01002525 help="Generate coverage report for unit tests, and"
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02002526 " tests and samples run in native_posix. Implies"
2527 " --enable_coverage")
Andrew Boie6acbe632015-07-17 12:03:52 -07002528
2529 return parser.parse_args()
2530
Anas Nashif3ba1d432017-12-05 15:28:44 -05002531
Andrew Boie6acbe632015-07-17 12:03:52 -07002532def log_info(filename):
Mazen NEIFER8f1dd3a2017-02-04 14:32:04 +01002533 filename = os.path.relpath(os.path.realpath(filename))
Andrew Boie6acbe632015-07-17 12:03:52 -07002534 if INLINE_LOGS:
Andrew Boie08ce5a52016-02-22 13:28:10 -08002535 info("{:-^100}".format(filename))
Andrew Boied01535c2017-01-10 13:15:02 -08002536
2537 try:
2538 with open(filename) as fp:
2539 data = fp.read()
2540 except Exception as e:
2541 data = "Unable to read log data (%s)\n" % (str(e))
2542
2543 sys.stdout.write(data)
2544 if log_file:
2545 log_file.write(data)
Andrew Boie08ce5a52016-02-22 13:28:10 -08002546 info("{:-^100}".format(filename))
Andrew Boie6acbe632015-07-17 12:03:52 -07002547 else:
Andrew Boie08ce5a52016-02-22 13:28:10 -08002548 info("\tsee: " + COLOR_YELLOW + filename + COLOR_NORMAL)
Andrew Boie6acbe632015-07-17 12:03:52 -07002549
Anas Nashif3ba1d432017-12-05 15:28:44 -05002550
Andrew Boie6acbe632015-07-17 12:03:52 -07002551def terse_test_cb(instances, goals, goal):
2552 total_tests = len(goals)
2553 total_done = 0
2554 total_failed = 0
2555
Andrew Boie08ce5a52016-02-22 13:28:10 -08002556 for k, g in goals.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07002557 if g.finished:
2558 total_done += 1
2559 if g.failed:
2560 total_failed += 1
2561
2562 if goal.failed:
2563 i = instances[goal.name]
Anas Nashif3ba1d432017-12-05 15:28:44 -05002564 info(
2565 "\n\n{:<25} {:<50} {}FAILED{}: {}".format(
2566 i.platform.name,
2567 i.test.name,
2568 COLOR_RED,
2569 COLOR_NORMAL,
2570 goal.reason))
Andrew Boie6acbe632015-07-17 12:03:52 -07002571 log_info(goal.get_error_log())
2572 info("")
2573
Anas Nashif3ba1d432017-12-05 15:28:44 -05002574 sys.stdout.write(
2575 "\rtotal complete: %s%4d/%4d%s %2d%% failed: %s%4d%s" %
2576 (COLOR_GREEN, total_done, total_tests, COLOR_NORMAL,
2577 int((float(total_done) / total_tests) * 100),
2578 COLOR_RED if total_failed > 0 else COLOR_NORMAL, total_failed,
2579 COLOR_NORMAL))
Andrew Boie6acbe632015-07-17 12:03:52 -07002580 sys.stdout.flush()
2581
Anas Nashif3ba1d432017-12-05 15:28:44 -05002582
Andrew Boie6acbe632015-07-17 12:03:52 -07002583def chatty_test_cb(instances, goals, goal):
2584 i = instances[goal.name]
2585
2586 if VERBOSE < 2 and not goal.finished:
2587 return
2588
Ruslan Mstoi33fa63e2018-05-29 14:35:34 +03002589 total_tests = len(goals)
2590 total_tests_width = len(str(total_tests))
2591 total_done = 0
2592
2593 for k, g in goals.items():
2594 if g.finished:
2595 total_done += 1
2596
Andrew Boie6acbe632015-07-17 12:03:52 -07002597 if goal.failed:
2598 status = COLOR_RED + "FAILED" + COLOR_NORMAL + ": " + goal.reason
2599 elif goal.finished:
2600 status = COLOR_GREEN + "PASSED" + COLOR_NORMAL
2601 else:
2602 status = goal.make_state
2603
Ruslan Mstoi33fa63e2018-05-29 14:35:34 +03002604 info("{:>{}}/{} {:<25} {:<50} {}".format(
2605 total_done, total_tests_width, total_tests, i.platform.name,
2606 i.test.name, status))
Andrew Boie6acbe632015-07-17 12:03:52 -07002607 if goal.failed:
2608 log_info(goal.get_error_log())
2609
Andrew Boiebbd670c2015-08-17 13:16:11 -07002610
2611def size_report(sc):
2612 info(sc.filename)
Andrew Boie73b4ee62015-10-07 11:33:22 -07002613 info("SECTION NAME VMA LMA SIZE HEX SZ TYPE")
Andrew Boie9882dcd2015-10-07 14:25:51 -07002614 for i in range(len(sc.sections)):
2615 v = sc.sections[i]
2616
Andrew Boie73b4ee62015-10-07 11:33:22 -07002617 info("%-17s 0x%08x 0x%08x %8d 0x%05x %-7s" %
2618 (v["name"], v["virt_addr"], v["load_addr"], v["size"], v["size"],
2619 v["type"]))
Andrew Boie9882dcd2015-10-07 14:25:51 -07002620
Andrew Boie73b4ee62015-10-07 11:33:22 -07002621 info("Totals: %d bytes (ROM), %d bytes (RAM)" %
Anas Nashif3ba1d432017-12-05 15:28:44 -05002622 (sc.rom_size, sc.ram_size))
Andrew Boiebbd670c2015-08-17 13:16:11 -07002623 info("")
2624
Anas Nashif3ba1d432017-12-05 15:28:44 -05002625
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03002626def generate_coverage(outdir, ignores):
2627 with open(os.path.join(outdir, "coverage.log"), "a") as coveragelog:
2628 coveragefile = os.path.join(outdir, "coverage.info")
2629 ztestfile = os.path.join(outdir, "ztest.info")
2630 subprocess.call(["lcov", "--capture", "--directory", outdir,
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01002631 "--rc", "lcov_branch_coverage=1",
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03002632 "--output-file", coveragefile], stdout=coveragelog)
2633 # We want to remove tests/* and tests/ztest/test/* but save tests/ztest
2634 subprocess.call(["lcov", "--extract", coveragefile,
Anas Nashif3ba1d432017-12-05 15:28:44 -05002635 os.path.join(ZEPHYR_BASE, "tests", "ztest", "*"),
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01002636 "--output-file", ztestfile,
2637 "--rc", "lcov_branch_coverage=1"], stdout=coveragelog)
2638
2639 if os.path.getsize(ztestfile) > 0:
2640 subprocess.call(["lcov", "--remove", ztestfile,
2641 os.path.join(ZEPHYR_BASE, "tests/ztest/test/*"),
2642 "--output-file", ztestfile,
2643 "--rc", "lcov_branch_coverage=1"],
2644 stdout=coveragelog)
2645 files = [coveragefile, ztestfile];
2646 else:
2647 files = [coveragefile];
2648
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03002649 for i in ignores:
Anas Nashif3ba1d432017-12-05 15:28:44 -05002650 subprocess.call(
2651 ["lcov", "--remove", coveragefile, i, "--output-file",
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01002652 coveragefile, "--rc", "lcov_branch_coverage=1"],
Anas Nashif3ba1d432017-12-05 15:28:44 -05002653 stdout=coveragelog)
Alberto Escolar Piedrasbc101c52018-02-11 09:33:55 +01002654
2655 ret = subprocess.call(["genhtml", "--legend", "--branch-coverage",
2656 "-output-directory",
2657 os.path.join(outdir, "coverage")] + files,
2658 stdout=coveragelog)
2659 if ret==0:
2660 info("HTML report generated: %s"%
2661 os.path.join(outdir, "coverage","index.html"));
Anas Nashif3ba1d432017-12-05 15:28:44 -05002662
Andrew Boiebbd670c2015-08-17 13:16:11 -07002663
Andrew Boie6acbe632015-07-17 12:03:52 -07002664def main():
Andrew Boie4b182472015-07-31 12:25:22 -07002665 start_time = time.time()
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03002666 global VERBOSE, INLINE_LOGS, JOBS, log_file
Anas Nashife10b6512017-12-30 13:01:45 -05002667 global options
2668 options = parse_arguments()
Andrew Boiebbd670c2015-08-17 13:16:11 -07002669
Alberto Escolar Piedrasc026c2e2018-06-21 09:30:20 +02002670 if options.coverage:
2671 options.enable_coverage = True
2672
Anas Nashife10b6512017-12-30 13:01:45 -05002673 if options.size:
2674 for fn in options.size:
Andrew Boie52fef672016-11-29 12:21:59 -08002675 size_report(SizeCalculator(fn, []))
Andrew Boiebbd670c2015-08-17 13:16:11 -07002676 sys.exit(0)
2677
Anas Nashif73440ea2018-02-19 10:57:03 -06002678
2679 if options.device_testing:
2680 if options.device_serial is None or len(options.platform) != 1:
2681 sys.exit(1)
2682
Anas Nashife10b6512017-12-30 13:01:45 -05002683 VERBOSE += options.verbose
2684 INLINE_LOGS = options.inline_logs
2685 if options.log_file:
2686 log_file = open(options.log_file, "w")
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03002687
Anas Nashife10b6512017-12-30 13:01:45 -05002688 if options.jobs:
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03002689 JOBS = options.jobs
Andrew Boie6acbe632015-07-17 12:03:52 -07002690
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03002691 # Decrease JOBS for Ninja, if jobs weren't explicitly set
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03002692 if options.ninja and not options.jobs:
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03002693 JOBS = int(JOBS * 0.75)
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03002694
Oleg Zhurakivskyy99aacd92018-08-17 15:02:28 +03002695 info("JOBS: %d" % JOBS);
Oleg Zhurakivskyyc97054c2018-08-17 14:56:05 +03002696
Anas Nashife10b6512017-12-30 13:01:45 -05002697 if options.subset:
2698 subset, sets = options.subset.split("/")
Anas Nashif035799f2017-05-13 21:31:53 -04002699 if int(subset) > 0 and int(sets) >= int(subset):
Anas Nashif3ba1d432017-12-05 15:28:44 -05002700 info("Running only a subset: %s/%s" % (subset, sets))
Anas Nashif035799f2017-05-13 21:31:53 -04002701 else:
Anas Nashife10b6512017-12-30 13:01:45 -05002702 error("You have provided a wrong subset value: %s." % options.subset)
Anas Nashif035799f2017-05-13 21:31:53 -04002703 return
2704
Anas Nashife10b6512017-12-30 13:01:45 -05002705 if os.path.exists(options.outdir) and not options.no_clean:
2706 info("Cleaning output directory " + options.outdir)
2707 shutil.rmtree(options.outdir)
Andrew Boie6acbe632015-07-17 12:03:52 -07002708
Anas Nashife10b6512017-12-30 13:01:45 -05002709 if not options.testcase_root:
2710 options.testcase_root = [os.path.join(ZEPHYR_BASE, "tests"),
Andrew Boie3d348712016-04-08 11:52:13 -07002711 os.path.join(ZEPHYR_BASE, "samples")]
2712
Anas Nashif37f9dc52018-02-23 08:53:46 -06002713 ts = TestSuite(options.board_root, options.testcase_root, options.outdir)
Anas Nashifbd166f42017-09-02 12:32:08 -04002714
Kumar Galac84235e2018-04-10 13:32:51 -05002715 if ts.load_errors:
2716 sys.exit(1)
2717
Anas Nashif75547e22018-02-24 08:32:14 -06002718 if options.list_tags:
2719 tags = set()
2720 for n,tc in ts.testcases.items():
2721 tags = tags.union(tc.tags)
2722
2723 for t in tags:
2724 print("- {}".format(t))
2725
2726 return
2727
Anas Nashif5d6e7eb2018-06-01 09:51:08 -05002728
2729 def export_tests(filename, tests):
2730 with open(filename, "wt") as csvfile:
2731 fieldnames = ['section', 'subsection', 'title', 'reference']
2732 cw = csv.DictWriter(csvfile, fieldnames, lineterminator=os.linesep)
2733 for test in tests:
2734 data = test.split(".")
2735 subsec = " ".join(data[1].split("_")).title()
2736 rowdict = {
2737 "section": data[0].capitalize(),
2738 "subsection": subsec,
2739 "title": test,
2740 "reference": test
2741 }
2742 cw.writerow(rowdict)
2743
2744 if options.export_tests:
2745 cnt = 0
2746 unq = []
2747 for n,tc in ts.testcases.items():
2748 for c in tc.cases:
2749 unq.append(c)
2750
2751 tests = sorted(set(unq))
2752 export_tests(options.export_tests, tests)
2753 return
2754
2755
Anas Nashifc0149cc2018-04-14 23:12:58 -05002756 if options.list_tests:
2757 cnt = 0
Anas Nashifa3abe962018-05-05 19:10:22 -05002758 unq = []
Anas Nashifc0149cc2018-04-14 23:12:58 -05002759 for n,tc in ts.testcases.items():
2760 for c in tc.cases:
Anas Nashifa3abe962018-05-05 19:10:22 -05002761 unq.append(c)
Anas Nashifc0149cc2018-04-14 23:12:58 -05002762
Anas Nashifa3abe962018-05-05 19:10:22 -05002763 for u in sorted(set(unq)):
2764 cnt = cnt + 1
2765 print(" - {}".format(u))
Anas Nashifc0149cc2018-04-14 23:12:58 -05002766 print("{} total.".format(cnt))
2767 return
2768
Anas Nashifbd166f42017-09-02 12:32:08 -04002769 discards = []
Anas Nashife10b6512017-12-30 13:01:45 -05002770 if options.load_tests:
2771 ts.load_from_file(options.load_tests)
Anas Nashifbd166f42017-09-02 12:32:08 -04002772 else:
Anas Nashif4f028882017-12-30 11:48:43 -05002773 discards = ts.apply_filters()
Andrew Boie6acbe632015-07-17 12:03:52 -07002774
Anas Nashife10b6512017-12-30 13:01:45 -05002775 if options.discard_report:
2776 ts.discard_report(options.discard_report)
Andrew Boie6acbe632015-07-17 12:03:52 -07002777
Anas Nashif30551f42018-01-12 21:56:59 -05002778 if VERBOSE > 1 and discards:
Anas Nashiff18e2ab2018-01-13 07:57:42 -05002779 # if we are using command line platform filter, no need to list every
2780 # other platform as excluded, we know that already.
2781 # Show only the discards that apply to the selected platforms on the
2782 # command line
2783
Andrew Boie08ce5a52016-02-22 13:28:10 -08002784 for i, reason in discards.items():
Anas Nashiff18e2ab2018-01-13 07:57:42 -05002785 if options.platform and i.platform.name not in options.platform:
2786 continue
Anas Nashif3ba1d432017-12-05 15:28:44 -05002787 debug(
2788 "{:<25} {:<50} {}SKIPPED{}: {}".format(
2789 i.platform.name,
2790 i.test.name,
2791 COLOR_YELLOW,
2792 COLOR_NORMAL,
2793 reason))
Andrew Boie6acbe632015-07-17 12:03:52 -07002794
Anas Nashif1a5bba72018-01-05 08:07:45 -05002795
Alberto Escolar Piedrasc6524ab2018-02-08 21:35:55 +01002796 def native_posix_and_unit_first(a, b):
2797 if a[0].startswith('native_posix') or a[0].startswith('unit_testing'):
Anas Nashif1a5bba72018-01-05 08:07:45 -05002798 return -1
Alberto Escolar Piedrasc6524ab2018-02-08 21:35:55 +01002799 if b[0].startswith('native_posix') or b[0].startswith('unit_testing'):
Anas Nashif1a5bba72018-01-05 08:07:45 -05002800 return 1
2801 return (a > b) - (a < b)
2802
2803 ts.instances = OrderedDict(sorted(ts.instances.items(),
Alberto Escolar Piedrasc6524ab2018-02-08 21:35:55 +01002804 key=cmp_to_key(native_posix_and_unit_first)))
Anas Nashifbd166f42017-09-02 12:32:08 -04002805
Anas Nashife10b6512017-12-30 13:01:45 -05002806 if options.save_tests:
2807 ts.run_report(options.save_tests)
Anas Nashifbd166f42017-09-02 12:32:08 -04002808 return
2809
Anas Nashife10b6512017-12-30 13:01:45 -05002810 if options.subset:
Anas Nashif1a5bba72018-01-05 08:07:45 -05002811
Anas Nashife10b6512017-12-30 13:01:45 -05002812 subset, sets = options.subset.split("/")
Anas Nashifbd166f42017-09-02 12:32:08 -04002813 total = len(ts.instances)
Anas Nashif035799f2017-05-13 21:31:53 -04002814 per_set = round(total / int(sets))
Anas Nashif3ba1d432017-12-05 15:28:44 -05002815 start = (int(subset) - 1) * per_set
Anas Nashif035799f2017-05-13 21:31:53 -04002816 if subset == sets:
2817 end = total
2818 else:
2819 end = start + per_set
2820
Anas Nashif3ba1d432017-12-05 15:28:44 -05002821 sliced_instances = islice(ts.instances.items(), start, end)
Anas Nashif035799f2017-05-13 21:31:53 -04002822 ts.instances = OrderedDict(sliced_instances)
2823
Andrew Boie6acbe632015-07-17 12:03:52 -07002824 info("%d tests selected, %d tests discarded due to filters" %
2825 (len(ts.instances), len(discards)))
2826
Anas Nashife10b6512017-12-30 13:01:45 -05002827 if options.dry_run:
Andrew Boie6acbe632015-07-17 12:03:52 -07002828 return
2829
2830 if VERBOSE or not TERMINAL:
Anas Nashif3ba1d432017-12-05 15:28:44 -05002831 goals = ts.execute(
2832 chatty_test_cb,
Anas Nashif37f9dc52018-02-23 08:53:46 -06002833 ts.instances)
Andrew Boie6acbe632015-07-17 12:03:52 -07002834 else:
Anas Nashif3ba1d432017-12-05 15:28:44 -05002835 goals = ts.execute(
2836 terse_test_cb,
Anas Nashif37f9dc52018-02-23 08:53:46 -06002837 ts.instances)
Andrew Boie08ce5a52016-02-22 13:28:10 -08002838 info("")
Andrew Boie6acbe632015-07-17 12:03:52 -07002839
Anas Nashife0a6a0b2018-02-15 20:07:24 -06002840 if options.detailed_report:
2841 ts.testcase_target_report(options.detailed_report)
2842
Daniel Leung7f850102016-04-08 11:07:32 -07002843 # figure out which report to use for size comparison
Anas Nashife10b6512017-12-30 13:01:45 -05002844 if options.compare_report:
2845 report_to_use = options.compare_report
2846 elif options.last_metrics:
Daniel Leung7f850102016-04-08 11:07:32 -07002847 report_to_use = LAST_SANITY
2848 else:
2849 report_to_use = RELEASE_DATA
2850
2851 deltas = ts.compare_metrics(report_to_use)
Andrew Boie6acbe632015-07-17 12:03:52 -07002852 warnings = 0
Anas Nashif424a3db2018-02-20 08:37:24 -06002853 if deltas and options.show_footprint:
Andrew Boieea7928f2015-08-14 14:27:38 -07002854 for i, metric, value, delta, lower_better in deltas:
Anas Nashife10b6512017-12-30 13:01:45 -05002855 if not options.all_deltas and ((delta < 0 and lower_better) or
Andrew Boieea7928f2015-08-14 14:27:38 -07002856 (delta > 0 and not lower_better)):
Andrew Boie6acbe632015-07-17 12:03:52 -07002857 continue
2858
Andrew Boieea7928f2015-08-14 14:27:38 -07002859 percentage = (float(delta) / float(value - delta))
Anas Nashife10b6512017-12-30 13:01:45 -05002860 if not options.all_deltas and (percentage <
2861 (options.footprint_threshold / 100.0)):
Andrew Boieea7928f2015-08-14 14:27:38 -07002862 continue
2863
Daniel Leung00525c22016-04-11 10:27:56 -07002864 info("{:<25} {:<60} {}{}{}: {} {:<+4}, is now {:6} {:+.2%}".format(
Andrew Boieea7928f2015-08-14 14:27:38 -07002865 i.platform.name, i.test.name, COLOR_YELLOW,
Anas Nashife10b6512017-12-30 13:01:45 -05002866 "INFO" if options.all_deltas else "WARNING", COLOR_NORMAL,
Andrew Boie829c0562015-10-13 09:44:19 -07002867 metric, delta, value, percentage))
Andrew Boie6acbe632015-07-17 12:03:52 -07002868 warnings += 1
2869
2870 if warnings:
2871 info("Deltas based on metrics from last %s" %
Anas Nashife10b6512017-12-30 13:01:45 -05002872 ("release" if not options.last_metrics else "run"))
Andrew Boie6acbe632015-07-17 12:03:52 -07002873
2874 failed = 0
Andrew Boie08ce5a52016-02-22 13:28:10 -08002875 for name, goal in goals.items():
Andrew Boie6acbe632015-07-17 12:03:52 -07002876 if goal.failed:
2877 failed += 1
Jaakko Hannikainenca505f82016-08-22 15:03:46 +03002878 elif goal.metrics.get("unrecognized"):
Andrew Boie73b4ee62015-10-07 11:33:22 -07002879 info("%sFAILED%s: %s has unrecognized binary sections: %s" %
2880 (COLOR_RED, COLOR_NORMAL, goal.name,
2881 str(goal.metrics["unrecognized"])))
2882 failed += 1
Andrew Boie6acbe632015-07-17 12:03:52 -07002883
Anas Nashife10b6512017-12-30 13:01:45 -05002884 if options.coverage:
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03002885 info("Generating coverage files...")
Anas Nashife10b6512017-12-30 13:01:45 -05002886 generate_coverage(options.outdir, ["tests/*", "samples/*"])
Jaakko Hannikainen54c90bc2016-08-31 14:17:03 +03002887
Anas Nashif0605fa32017-05-07 08:51:02 -04002888 duration = time.time() - start_time
Andrew Boie4b182472015-07-31 12:25:22 -07002889 info("%s%d of %d%s tests passed with %s%d%s warnings in %d seconds" %
Anas Nashif3ba1d432017-12-05 15:28:44 -05002890 (COLOR_RED if failed else COLOR_GREEN, len(goals) - failed,
2891 len(goals), COLOR_NORMAL, COLOR_YELLOW if warnings else COLOR_NORMAL,
2892 warnings, COLOR_NORMAL, duration))
Andrew Boie6acbe632015-07-17 12:03:52 -07002893
Anas Nashife10b6512017-12-30 13:01:45 -05002894 if options.testcase_report:
2895 ts.testcase_report(options.testcase_report)
2896 if not options.no_update:
Anas Nashif4f028882017-12-30 11:48:43 -05002897 ts.testcase_xunit_report(LAST_SANITY_XUNIT, duration)
Andrew Boie6acbe632015-07-17 12:03:52 -07002898 ts.testcase_report(LAST_SANITY)
Anas Nashife10b6512017-12-30 13:01:45 -05002899 if options.release:
Andrew Boie6acbe632015-07-17 12:03:52 -07002900 ts.testcase_report(RELEASE_DATA)
Inaky Perez-Gonzalez9a36cb62016-11-29 10:43:40 -08002901 if log_file:
2902 log_file.close()
Anas Nashife10b6512017-12-30 13:01:45 -05002903 if failed or (warnings and options.warnings_as_errors):
Andrew Boie6acbe632015-07-17 12:03:52 -07002904 sys.exit(1)
2905
Anas Nashif3ba1d432017-12-05 15:28:44 -05002906
Andrew Boie6acbe632015-07-17 12:03:52 -07002907if __name__ == "__main__":
2908 main()