This page describes the Matter project's functional testing approach for Mbed applications. It allows you to better understand the use of tools and frameworks for target validation.
According to the Matter project design idea:
“The ability to run tests on actual and emulated hardware is paramount in embedded projects. CHIP is no exception. We want on-device testing to be a first class goal of CHIP architecture. On-device testing requirements apply both to Continuous Integration testing for main CHIP software stack development and to eventual CHIP product certification.”
However, functional testing requires a host machine to run and control the testing program. Connection and data exchange between the host machine and the device under test is necessary. Various communication interfaces and protocols are used for this purpose (i.e serial port, network communication via Ethernet, or WiFi).
The common functional test scenario involves the following steps:
Matter Arm Mbed OS Integration Tests happens at system level on real hardware. The host machine running the tests should be connected to the board under test via serial port. The tests send a succession of commands to the connected boards or trigger external tools action that changes the device state and asserts the results reported by the hardware.
For example, here is a test validating that WiFi connection works:
Preparation: The device is connected to the host via serial port and executes the shell example application.
The test:
The Mbed integration tests are coded in Python and use the pyTest framework. This also allows for easy integration with external CHIP tools such as device-controller.
Python CHIP Controller and RPC console are required for the correct run integration tests.
The first step check out the Matter repository and sync submodules using the following command:
$ git submodule update --init
Building the example application requires the use of ARM Mbed-OS sources and the arm-none-gnu-eabi toolchain.
The Cypress OpenOCD package is required for flashing purpose. Install the Cypress OpenOCD and set env var OPENOCD_PATH before calling the flashing script.
cd ~ wget https://github.com/Infineon/openocd/releases/download/release-v4.3.0/openocd-4.3.0.1746-linux.tar.gz tar xzvf openocd-4.3.0.1746-linux.tar.gz export OPENOCD_PATH=$HOME/openocd
Some additional packages may be needed, depending on selected build target and its requirements.
The VSCode devcontainer has these components pre-installed. Using the VSCode devcontainer is the recommended way to interact with Arm Mbed-OS port of the Matter Project.
Please read this README.md for more information about using VSCode in container.
To initialize the development environment, download all registered sub-modules and activate the environment:
$ source ./scripts/bootstrap.sh $ source ./scripts/activate.sh
If packages are already installed then you just need to activate the development environment:
$ source ./scripts/activate.sh
Python CHIP Controller and RPC console are required for the correct run integration tests. Build and install them inside the development environment.
For building and installing Python CHIP Controller please visit Python CHIP Controller
For building and installing RPC console please visit CHIP RPC CONSOLE
The way of running the WiFi access point depends on the platform on which the tests are run. Choose the best one. Just remember the network credentials ssid and password.
There is a special script for the easy Mbed board mounting. It adds a static entry in fstab with a new connected device.
Then you can just need to call sudo mount -a to mount the device.
To add the new Mbed device you should:
bash src/scripts/tests/mbed/mount_new_mbed_device.shsudo mount -aThat's all. The new device is ready for testing.
Preparation of the device for testing should consist of two steps:
For more information how to build and flash example application please visit their documentation.
Mbed integration tests can be configured by PyTest command line arguments. Every test call may contain a specific test configuration. The list of supported parameters:
[Common]
[Test specific]
To run Mbed integration tests execute the pytest command with the arguments mentioned above. For example:
pytest --platforms=CY8CPROTO_062_4343W --network=$AP_SSID:$AP_PASSWORD ... src/test_driver/mbed/integration_tests/{APP_NAME}/test_app.py
The Mbed integration testes cases are divided into separate directory depends on the testing Matter application:
For more details on how to run tests using PyTest see: PyTest doc
Pytest markers have been added to run a specific set of tests:
Adding -rAv arguments to Pytest cause that short tests summary is on the output.
For example:
pytest -rAv simple_test.py
Output:
=================================================================================================================== short test summary info ==================================================================================================================== PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_smoke_test PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_help_check PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_log_check PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_rand_check PASSED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_base64_encode_decode FAILED CHIP/src/test_driver/mbed/integration_tests/shell/test_app.py::test_wifi_mode - AssertionError: assert 'true' == 'false' ===================================================================================================== 1 failed, 4 passed, 7 warnings in 20.71s (0:03:29) =====================================================================================================
There is also an option to save test results to HTML file. Adding --html=<file_name.html> arguments to Pytest cause that all test results are saved to HTML file and can be open in your browser.
For example:
pytest --html=result.html simple_test.py