tree: 5deac814d035c8f7f8ba83bb081bf3bc433a192d [path history] [tgz]
  1. common/
  2. lighting-app/
  3. lock-app/
  4. pigweed-app/
  5. shell/
  6. unit-tests/
  7. conftest.py
  8. pytest.ini
  9. README.md
  10. test_set.in
src/test_driver/mbed/integration_tests/README.md

ARM Mbed-OS logo

This page describes the Matter project's functional testing approach for Mbed applications. It allows you to better understand the use of tools and frameworks for target validation.

Overview

According to the Matter project design idea:

“The ability to run tests on actual and emulated hardware is paramount in embedded projects. CHIP is no exception. We want on-device testing to be a first class goal of CHIP architecture. On-device testing requirements apply both to Continuous Integration testing for main CHIP software stack development and to eventual CHIP product certification.”

However, functional testing requires a host machine to run and control the testing program. Connection and data exchange between the host machine and the device under test is necessary. Various communication interfaces and protocols are used for this purpose (i.e serial port, network communication via Ethernet, or WiFi).

The common functional test scenario involves the following steps:

  • Get binaries image of testing application (build directly or download CI artifacts)
  • Flash binary image to the device under test
  • Configuration functional tests
  • Run testing
  • Collecting and processing the results

Matter Arm Mbed OS Integration Tests happens at system level on real hardware. The host machine running the tests should be connected to the board under test via serial port. The tests send a succession of commands to the connected boards or trigger external tools action that changes the device state and asserts the results reported by the hardware.

For example, here is a test validating that WiFi connection works:

Preparation: The device is connected to the host via serial port and executes the shell example application.

The test:

  1. Checking if the device is connected
  2. Sending the command connecting with network credentials
  3. Check device response
  4. Check the device IP address

The Mbed integration tests are coded in Python and use the pyTest framework. This also allows for easy integration with external CHIP tools such as device-controller.

Python CHIP Controller and RPC console are required for the correct run integration tests.

Setup

Environment setup

The first step check out the Matter repository and sync submodules using the following command:

$ git submodule update --init

Building the example application requires the use of ARM Mbed-OS sources and the arm-none-gnu-eabi toolchain.

The Cypress OpenOCD package is required for flashing purpose. Install the Cypress OpenOCD and set env var OPENOCD_PATH before calling the flashing script.

cd ~
wget https://github.com/Infineon/openocd/releases/download/release-v4.3.0/openocd-4.3.0.1746-linux.tar.gz
tar xzvf openocd-4.3.0.1746-linux.tar.gz
export OPENOCD_PATH=$HOME/openocd

Some additional packages may be needed, depending on selected build target and its requirements.

The VSCode devcontainer has these components pre-installed. Using the VSCode devcontainer is the recommended way to interact with Arm Mbed-OS port of the Matter Project.

Please read this README.md for more information about using VSCode in container.

To initialize the development environment, download all registered sub-modules and activate the environment:

$ source ./scripts/bootstrap.sh
$ source ./scripts/activate.sh

If packages are already installed then you just need to activate the development environment:

$ source ./scripts/activate.sh

Tools installation

Python CHIP Controller and RPC console are required for the correct run integration tests. Build and install them inside the development environment.

For building and installing Python CHIP Controller please visit Python CHIP Controller

For building and installing RPC console please visit CHIP RPC CONSOLE

Setup WiFi access point

The way of running the WiFi access point depends on the platform on which the tests are run. Choose the best one. Just remember the network credentials ssid and password.

Mount the device

There is a special script for the easy Mbed board mounting. It adds a static entry in fstab with a new connected device.

Then you can just need to call sudo mount -a to mount the device.

To add the new Mbed device you should:

  1. Run script bash src/scripts/tests/mbed/mount_new_mbed_device.sh
  2. Plug the device
  3. Run sudo mount -a

That's all. The new device is ready for testing.

Prepare device under test

Preparation of the device for testing should consist of two steps:

  • prepare binary image - building an right application from sources
  • flash binary image to device

For more information how to build and flash example application please visit their documentation.

Tests configuration

Mbed integration tests can be configured by PyTest command line arguments. Every test call may contain a specific test configuration. The list of supported parameters:

[Common]

  • platforms - list of platforms that can be used to run the tests. Platforms are separated by a comma
  • binaries - platform and associated binary in the form platform:binary. Multiple values are separated by a comma
  • serial_inter_byte_delay - time in second between two bytes sent on the serial line (accepts floats), default=None
  • serial_baudrate - baudrate of the serial port used, default=115200

[Test specific]

  • network - WiFi network credentials to which we want to connect device. Format network_ssid:network_password

Test run

To run Mbed integration tests execute the pytest command with the arguments mentioned above. For example:

pytest --platforms=CY8CPROTO_062_4343W --network=$AP_SSID:$AP_PASSWORD ... src/test_driver/mbed/integration_tests/{APP_NAME}/test_app.py

The Mbed integration testes cases are divided into separate directory depends on the testing Matter application:

  • shell - testing shell commands and check base device functionalities
  • lock-app - testing WiFi provisioning and execute ZCL command to control lock, use RPC client to run base device functionalities, control lock and trigger some button actions
  • lighting-app - testing WiFi provisioning and execute ZCL command to control light, use RPC client to run base device functionalities, control light and trigger some button actions
  • pigweed-app - use RPC client to send echo message and receive the response
  • unit-tests - check unit-tests result

For more details on how to run tests using PyTest see: PyTest doc

Pytest markers have been added to run a specific set of tests:

  • smoketest - check base communication and correct launch of the application

Tests results

Adding -rAv arguments to Pytest cause that short tests summary is on the output.

For example:

pytest -rAv simple_test.py

Output:

=================================================================================================================== short test summary info ====================================================================================================================
PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_smoke_test
PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_help_check
PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_log_check
PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_rand_check
PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_base64_encode_decode
FAILED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_wifi_mode - AssertionError: assert 'true' == 'false'
===================================================================================================== 1 failed, 4 passed, 7 warnings in 20.71s (0:03:29) =====================================================================================================

There is also an option to save test results to HTML file. Adding --html=<file_name.html> arguments to Pytest cause that all test results are saved to HTML file and can be open in your browser.

For example:

pytest --html=result.html simple_test.py