CHIP Test Suites

This directory contains a set of tests describing interactions between nodes, and more specifically between a controller and a controllee.

This test set is written in a high level language before being automatically converted into a language understandable by the different controllers.

Controllers

There are currently 4 implementations of the CHIP device controller. These different implementations share a common configuration file describing the supported functionalities in terms of clusters, commands and attributes: controllers-clusters.zap

ControllersTesting
POSIX CLIy
Darwin CLIy
iOSy
Pythonn
Androidn

For more information on the different implementations of the CHIP device controller, see README.md

Automatic conversion process

The process of automatic conversion of test files depends on the ZAP tool.

Each of the CHIP device controller implementations uses a dedicated template translating the tests into an appropriate format.

ControllersTemplate
POSIX CLIexamples/chip-tool/templates/partials/test_cluster.zapt
Darwin CLIexamples/darwin-framework-tool/templates/partials/test_cluster.zapt
iOSsrc/darwin/Framework/CHIP/templates/partials/test_cluster.zapt
Python
Android

These dedicated templates share a common script which augments the test file with the content of the ZAP database built from the definition files of the specification.

Lists

This common script exposes the result of the analysis in the form of multiple lists accessible from the dedicated template files.

These lists contain elements of which certain properties are directly configurable through the source test file, but also additional properties added during the analysis.

List: chip_tests

{{chip_tests}} takes as parameter a list of test files and returns an iterable list of objects.

Each element of the list is an object with the following properties:

Property: test

NameDescriptionRequired
nameName of the test set in human readable formatYes
configDefault configuration that is inherited by each of the tests in this setYes
testsThe set of tests. It is possible to iterate over it using {{chip_test_items}}Yes

Property: config

NameDescriptionRequired
clusterName of the cluster listed in Cluster NamesNo
endpointEndpoint Identifier that will be targeted by default by each of the tests in this setNo
identityName of the controller to perform tests with. alpha, beta, and gamma are the only ones supportedNo
{variable_name}Name of test variable to use within test.No
Property: {variable_name}
NameDescription
{variable_name}Name of test variable to use within test.
typeMacro type from ZAP atomics

Additionally, the object exhibits the following autogenerated properties:

NameDescription
filenameName of the source file from which the test set was built
totalTestsTotal number of tests contained in this set

List: chip_tests_items

{{chip_tests_items}} can be used inside a {{chip_tests}} block and iterates over the test set.

Each element of the list is an object with the following configurable properties:

Property: tests

NameDescriptionRequired
labelTest nameYes
disabledAllows to deactivate a testNo
commandName of the command to execute. The command can be any of the commands available for the cluster, or a special command.No
attributeIf the command is a special attribute command, then this property contains the name of the attribute targeted by the command.No
optionalAllows you to specify that a test is optional. An optional test passes if the controllee does not support the requested command or if the command is supported, and the specified conditions are validated.No
clusterName from Cluster names . If not specified, the default configuration is used.No
endpointThe identifier of the target endpoint. If not specified, the default configuration is used.No
argumentsA list of arguments to pass to the command. It is possible to iterate over it using {{chip_test_item_parameters}}No
responseA list of expected results or result constraints. It is possible to iterate over it using {{chip_test_item_response_parameters}}No
PICSProtocol Implementation Conformance Statement. A conditional flag that this determines whether a test step should run.No
timedInteractionTimeoutMsSets a timeout for Writing to attribute and sending a command.No
Property: arguments
NameDescriptionRequired
valuesA list of argumentsYes
Property: values
NameDescriptionRequired
nameThe name of the argument to be usedNo
valueThe value of the argument to be used within test stepYes
Property: response
NameDescriptionRequired
valuesA list of values to expect on response. See Property: valueNo (If other provided)
valueThe values expected on the the response from the commandNo (If other provided)
errorError code that is expected from responseNo (If other provided)
saveAsSave a value to a variable name for later use. Value above required. ExampleNo
constraintsConstraints to check for on response from the commandNo
Property: constraints
NameDescriptionRequired
hasValueIf true, must have value. If false, must not have value.No (If other provided)
minValueMinimum value to expect from the command response.No (If other provided)
maxValueMaximum value to expect from the command response.No (If other provided)
notValueValidate the the value is not what is provided.No (If other provided)
minLengthMinimum length of the response parameter.No (If other provided)
maxLengthMaximum length of the string parameter.No (If other provided)
startsWithCondition is which will validate what a string starts with.No (If other provided)
endsWithCondition is which will validate what a string ends with.No (If other provided)
isLowerCaseValidates if the char_string is lower case.No (If other provided)
isUpperCaseValidates if the char_string is upper case.No (If other provided)
isHexStringChecks whether the char_string is a hex string.No (If other provided)

Note: The hasValue constraint is only applied to optional fields. The other constraints are ignored for optional fields that do not have a value.

Additionally, the object exhibits the following autogenerated properties:

NameDescription
indexOverall test index

List: chip_tests_item_parameters

{{chip_tests_item_parameters}} can be used inside a {{chip_test_items}} block and iterates over the test arguments for the target command.

This object contains all the properties contained in the ZAP database related to the command or the attribute in the case of a special attribute command.

It is increased by the value to be used contained in the test file.

List: chip_tests_item_response_parameters

{{chip_tests_item_response_parameters}} can be used inside a {{chip_test_items}} block and iterates over the test response arguments for the target command.

This object contains all the properties contained in the ZAP database related to the response or the attribute response in the case of a special attribute command.

It is increased by the expected value or the constraints to be used contained in the test file.

List: chip_tests_pics

{{chip_tests_pics}} can be used to get the collection of all PICS for a given test.

List: chip_tests_config

{{chip_tests_config}} can be used to get the collection of user defined variables in the config. See ZAP Tests Config below

This object contains {{name}} {{definedValue}} and {{type}}. (chip_tests_config_has definedValue) can be used to determine if there is a value set for a particular user defined variable.

NameDescription
nameThe name of the user defined variable
typeZAP Atomic Types Macro
definedValueThe value of the user define variable

YAML Test Definition

YAML is used to generate tests into a programming language understood by commissioners, controllers, or simulated devices. There is a define set of keyword/values that are used to define test behavior.

YAML is the high level language chosen to generate test against a controller or a controllee. Test Definition Example

Top level key to define a YAML test: test

Location of Test Definitions

  • Simulated README

  • Simulated tests: tests.js

  • chip-tool(darwin) tests: tests.js

  • chip-tool tests: tests.js

    // Manual test definitions. Does not run in CI.
    function  getManualTests();
    
    // Test definition that run in CI.
    function getTests();
    

PICS Usage

Required Files

Example PICS Command

chip-tool tests Test 1 --PICS TestPICS.txt

Examples

YAML Examples

ZAP Example

Required Files

Generate Example Script

The following script will generate an output file TestGenExample.out in src/app/tests/suites/examples/out. The directory and the file will be created. This is a basic demonstration on how tests are generated.

src/app/tests/suites/examples/gen_readme_example.sh

Index

NameSizeMacroIDDescriptionC type
no_data0NO_DATA0x00No datauint8_t *
boolean1BOOLEAN0x10Booleanuint8_t
bitmap81BITMAP80x188-bit bitmapuint8_t
bitmap162BITMAP160x1916-bit bitmapuint16_t
bitmap243BITMAP240x1A24-bit bitmapuint32_t
bitmap324BITMAP320x1B32-bit bitmapuint32_t
bitmap648BITMAP640x1F64-bit bitmapuint8_t *
int8u1INT8U0x20Unsigned 8-bit integeruint8_t
int16u2INT16U0x21Unsigned 16-bit integeruint16_t
int24u3INT24U0x22Unsigned 24-bit integeruint32_t
int32u4INT32U0x23Unsigned 32-bit integeruint32_t
int40u5INT40U0x24Unsigned 40-bit integeruint8_t *
int48u6INT48U0x25Unsigned 48-bit integeruint8_t *
int56u7INT56U0x26Unsigned 56-bit integeruint8_t *
int64u8INT64U0x27Unsigned 64-bit integeruint8_t *
int8s1INT8S0x28Signed 8-bit integerint8_t
int16s2INT16S0x29Signed 16-bit integerint16_t
int24s3INT24S0x2ASigned 24-bit integerint32_t
int32s4INT32S0x2BSigned 32-bit integerint32_t
int40s5INT40S0x2CSigned 40-bit integerint8_t *
int48s6INT48S0x2DSigned 48-bit integerint8_t *
int56s7INT56S0x2ESigned 56-bit integerint8_t *
int64s8INT64S0x2FSigned 64-bit integerint8_t *
enum81ENUM80x308-bit enumerationuint8_t
enum162ENUM160x3116-bit enumerationuint16_t
single4SINGLE0x39Single precisionuint8_t *
double8DOUBLE0x3ADouble precisionuint8_t *
octet_stringOCTET_STRING0x41Octet Stringuint8_t *
char_stringCHAR_STRING0x42Character Stringuint8_t *
long_octet_stringLONG_OCTET_STRING0x43Long Octet Stringuint8_t *
long_char_stringLONG_CHAR_STRING0x44Long Character Stringuint8_t *
arrayARRAY0x48Listuint8_t *
structSTRUCT0x4CStructureuint8_t *
tod4TOD0xE0Time of dayuint8_t *
date4DATE0xE1Dateuint32_t
utc4UTC0xE2UTC Timeuint8_t *
epoch_us8EPOCH_US0xE3Epoch Microsecondsuint8_t *
epoch_s4EPOCH_S0xE4Epoch Secondsuint8_t *
systime_us8SYSTIME_US0xE5System Time Microsecondsuint8_t *
percent1PERCENT0xE6Percentage units 1%uint8_t *
percent100ths2PERCENT100THS0xE7Percentage units 0.01%uint8_t *
cluster_id4CLUSTER_ID0xE8Cluster IDuint16_t
attrib_id4ATTRIB_ID0xE9Attribute IDuint8_t *
field_id4FIELD_ID0xEAField IDuint8_t *
event_id4EVENT_ID0xEBEvent IDuint8_t *
command_id4COMMAND_ID0xECCommand IDuint8_t *
action_id1ACTION_ID0xEDAction IDuint8_t *
trans_id4TRANS_ID0xEFTransaction IDuint8_t *
node_id8NODE_ID0xF0Node IDuint8_t *
vendor_id2VENDOR_ID0xF1Vendor IDuint8_t *
devtype_id4DEVTYPE_ID0xF2Device Type IDuint8_t *
fabric_id8FABRIC_ID0xF3Fabric IDuint8_t *
group_id2GROUP_ID0xF4Group IDuint8_t *
status2STATUS0xF5Status Codeuint8_t *
data_ver4DATA_VER0xF6Data Versionuint_ver_t
event_no8EVENT_NO0xF7Event Numberuint8_t *
endpoint_no2ENDPOINT_NO0xF8Endpoint Numberuint8_t *
fabric_idx1FABRIC_IDX0xF9Fabric Indexuint8_t *
ipadrIPADR0xFAIP Addressuint8_t *
ipv4adr4IPV4ADR0xFBIPv4 Addressuint8_t *
ipv6adr16IPV6ADR0xFCIPv6 Addressuint8_t *
ipv6preIPV6PRE0xFDIPv6 Prefixuint8_t *
hwadrHWADR0xFEHardware Addressuint8_t *
unknown0UNKNOWN0xFFUnknownuint8_t *
PowerConfigurationAdministratorCommissioningDissolvedOxygenConcentrationMeasurement
DeviceTemperatureConfigurationOperationalCredentialsBromateConcentrationMeasurement
IdentifyGroupKeyManagementChloraminesConcentrationMeasurement
GroupsFixedLabelChlorineConcentrationMeasurement
ScenesUserLabelFecalColiformAndEColiConcentrationMeasurement
OnOffProxyConfigurationFluorideConcentrationMeasurement
OnOffSwitchConfigurationProxyDiscoveryHaloaceticAcidsConcentrationMeasurement
LevelControlProxyValidTotalTrihalomethanesConcentrationMeasurement
AlarmsBooleanStateTotalColiformBacteriaConcentrationMeasurement
TimeModeSelectTurbidityConcentrationMeasurement
BinaryInputBasicShadeConfigurationCopperConcentrationMeasurement
PowerProfileDoorLockLeadConcentrationMeasurement
ApplianceControlWindowCoveringManganeseConcentrationMeasurement
PulseWidthModulationBarrierControlSulfateConcentrationMeasurement
DescriptorPumpConfigurationAndControlBromodichloromethaneConcentrationMeasurement
BindingThermostatBromoformConcentrationMeasurement
AccessControlFanControlChlorodibromomethaneConcentrationMeasurement
PollControlDehumidificationControlChloroformConcentrationMeasurement
ActionsThermostatUserInterfaceConfigurationSodiumConcentrationMeasurement
BasicColorControlIasZone
OtaSoftwareUpdateProviderBallastConfigurationIasAce
OtaSoftwareUpdateRequestorIlluminanceMeasurementIasWd
LocalizationConfigurationTemperatureMeasurementWakeOnLan
TimeFormatLocalizationPressureMeasurementChannel
UnitLocalizationFlowMeasurementTargetNavigator
PowerSourceConfigurationRelativeHumidityMeasurementMediaPlayback
PowerSourceOccupancySensingMediaInput
GeneralCommissioningCarbonMonoxideConcentrationMeasurementLowPower
NetworkCommissioningCarbonDioxideConcentrationMeasurementKeypadInput
DiagnosticLogsEthyleneConcentrationMeasurementContentLauncher
GeneralDiagnosticsEthyleneOxideConcentrationMeasurementAudioOutput
SoftwareDiagnosticsHydrogenConcentrationMeasurementApplicationLauncher
ThreadNetworkDiagnosticsHydrogenSulphideConcentrationMeasurementApplicationBasic
WiFiNetworkDiagnosticsNitricOxideConcentrationMeasurementAccountLogin
EthernetNetworkDiagnosticsNitrogenDioxideConcentrationMeasurementTestCluster
TimeSynchronizationOxygenConcentrationMeasurementMessaging
BridgedDeviceBasicOzoneConcentrationMeasurementApplianceIdentification
SwitchSulfurDioxideConcentrationMeasurementMeterIdentification