CHIP Test Suites

• CHIP Test Suites
• Controllers
• Automatic conversion process
• List
  • chip_tests
    • Property: test
    • Property: config
    • Property: {variable_name}
  • chip_tests_items
    • Property: tests
    • Property: arguments
    • Property: values
    • Property: response
    • Property: constraints
  • chip_tests_item_parameters
  • chip_tests_item_response_parameters
  • chip_tests_pics
• YAML Test Definition
• PICS Usage
• Examples
  • YAML Examples
  • ZAP Example
• Index

This directory contains a set of tests describing interactions between nodes, and more specifically between a controller and a controllee.

This test set is written in a high level language before being automatically converted into a language understandable by the different controllers.

Controllers

There are currently 4 implementations of the CHIP device controller. These different implementations share a common configuration file describing the supported functionalities in terms of clusters, commands and attributes: controllers-clusters.zap

Controllers	Testing
POSIX CLI	y
Darwin CLI	y
iOS	y
Python	n
Android	n

For more information on the different implementations of the CHIP device controller, see README.md

Automatic conversion process

The process of automatic conversion of test files depends on the ZAP tool.

Each of the CHIP device controller implementations uses a dedicated template translating the tests into an appropriate format.

Controllers	Template
POSIX CLI	examples/chip-tool/templates/partials/test_cluster.zapt
Darwin CLI	examples/darwin-framework-tool/templates/partials/test_cluster.zapt
iOS	src/darwin/Framework/CHIP/templates/partials/test_cluster.zapt
Python
Android

These dedicated templates share a common script which augments the test file with the content of the ZAP database built from the definition files of the specification.

Lists

This common script exposes the result of the analysis in the form of multiple lists accessible from the dedicated template files.

These lists contain elements of which certain properties are directly configurable through the source test file, but also additional properties added during the analysis.

List: chip_tests

{{chip_tests}} takes as parameter a list of test files and returns an iterable list of objects.

Each element of the list is an object with the following properties:

Property: test

Name	Description	Required
name	Name of the test set in human readable format	Yes
config	Default configuration that is inherited by each of the tests in this set	Yes
tests	The set of tests. It is possible to iterate over it using {{chip_test_items}}	Yes

Property: config

Name	Description	Required
cluster	Name of the cluster listed in Cluster Names	No
endpoint	Endpoint Identifier that will be targeted by default by each of the tests in this set	No
identity	Name of the controller to perform tests with. alpha, beta, and gamma are the only ones supported	No
{variable_name}	Name of test variable to use within test.	No

Property: {variable_name}

Name	Description
{variable_name}	Name of test variable to use within test.
type	Macro type from ZAP atomics

Additionally, the object exhibits the following autogenerated properties:

Name	Description
filename	Name of the source file from which the test set was built
totalTests	Total number of tests contained in this set

List: chip_tests_items

{{chip_tests_items}} can be used inside a {{chip_tests}} block and iterates over the test set.

Each element of the list is an object with the following configurable properties:

Property: tests

Name	Description	Required
label	Test name	Yes
disabled	Allows to deactivate a test	No
command	Name of the command to execute. The command can be any of the commands available for the cluster, or a special command.	No
attribute	If the command is a special attribute command, then this property contains the name of the attribute targeted by the command.	No
optional	Allows you to specify that a test is optional. An optional test passes if the controllee does not support the requested command or if the command is supported, and the specified conditions are validated.	No
cluster	Name from Cluster names . If not specified, the default configuration is used.	No
endpoint	The identifier of the target endpoint. If not specified, the default configuration is used.	No
arguments	A list of arguments to pass to the command. It is possible to iterate over it using {{chip_test_item_parameters}}	No
response	A list of expected results or result constraints. It is possible to iterate over it using {{chip_test_item_response_parameters}}	No
PICS	Protocol Implementation Conformance Statement. A conditional flag that this determines whether a test step should run.	No
timedInteractionTimeoutMs	Sets a timeout for Writing to attribute and sending a command.	No

Property: arguments

Name	Description	Required
values	A list of arguments	Yes

Property: values

Name	Description	Required
name	The name of the argument to be used	No
value	The value of the argument to be used within test step	Yes

Property: response

Name	Description	Required
values	A list of values to expect on response. See Property: value	No (If other provided)
value	The values expected on the the response from the command	No (If other provided)
error	Error code that is expected from response	No (If other provided)
saveAs	Save a value to a variable name for later use. Value above required. Example	No
constraints	Constraints to check for on response from the command	No

Property: constraints

Name	Description	Required
hasValue	If true, must have value. If false, must not have value.	No (If other provided)
minValue	Minimum value to expect from the command response.	No (If other provided)
maxValue	Maximum value to expect from the command response.	No (If other provided)
notValue	Validate the the value is not what is provided.	No (If other provided)
minLength	Minimum length of the response parameter.	No (If other provided)
maxLength	Maximum length of the string parameter.	No (If other provided)
startsWith	Condition is which will validate what a string starts with.	No (If other provided)
endsWith	Condition is which will validate what a string ends with.	No (If other provided)
isLowerCase	Validates if the char_string is lower case.	No (If other provided)
isUpperCase	Validates if the char_string is upper case.	No (If other provided)
isHexString	Checks whether the char_string is a hex string.	No (If other provided)

Note: The hasValue constraint is only applied to optional fields. The other constraints are ignored for optional fields that do not have a value.

Additionally, the object exhibits the following autogenerated properties:

Name	Description
index	Overall test index

List: chip_tests_item_parameters

{{chip_tests_item_parameters}} can be used inside a {{chip_test_items}} block and iterates over the test arguments for the target command.

This object contains all the properties contained in the ZAP database related to the command or the attribute in the case of a special attribute command.

It is increased by the value to be used contained in the test file.

List: chip_tests_item_response_parameters

{{chip_tests_item_response_parameters}} can be used inside a {{chip_test_items}} block and iterates over the test response arguments for the target command.

This object contains all the properties contained in the ZAP database related to the response or the attribute response in the case of a special attribute command.

It is increased by the expected value or the constraints to be used contained in the test file.

List: chip_tests_pics

{{chip_tests_pics}} can be used to get the collection of all PICS for a given test.

List: chip_tests_config

{{chip_tests_config}} can be used to get the collection of user defined variables in the config. See ZAP Tests Config below

This object contains {{name}} {{definedValue}} and {{type}}. (chip_tests_config_has definedValue) can be used to determine if there is a value set for a particular user defined variable.

Name	Description
name	The name of the user defined variable
type	ZAP Atomic Types Macro
definedValue	The value of the user define variable

YAML Test Definition

YAML is used to generate tests into a programming language understood by commissioners, controllers, or simulated devices. There is a define set of keyword/values that are used to define test behavior.

YAML is the high level language chosen to generate test against a controller or a controllee. Test Definition Example

Top level key to define a YAML test: test

Location of Test Definitions

• Simulated README

• Simulated tests: tests.js

• chip-tool(darwin) tests: tests.js

• chip-tool tests: tests.js

  // Manual test definitions. Does not run in CI.
  function  getManualTests();
  
  // Test definition that run in CI.
  function getTests();
  

PICS Usage

Required Files

• PICS Test YAML
• TestPICS

Example PICS Command

chip-tool tests Test 1 --PICS TestPICS.txt

Examples

YAML Examples

• Basic YAML Config Example
• YAML Response Example
• YAML Tests Example
• YAML Config Variables Example

ZAP Example

Required Files

• ZAP Example
• Test1
• Test2
• Test3

Generate Example Script

The following script will generate an output file TestGenExample.out in src/app/tests/suites/examples/out. The directory and the file will be created. This is a basic demonstration on how tests are generated.

src/app/tests/suites/examples/gen_readme_example.sh

Index

Name	Size	Macro	ID	Description	C type
no_data	0	NO_DATA	0x00	No data	uint8_t *
boolean	1	BOOLEAN	0x10	Boolean	uint8_t
bitmap8	1	BITMAP8	0x18	8-bit bitmap	uint8_t
bitmap16	2	BITMAP16	0x19	16-bit bitmap	uint16_t
bitmap24	3	BITMAP24	0x1A	24-bit bitmap	uint32_t
bitmap32	4	BITMAP32	0x1B	32-bit bitmap	uint32_t
bitmap64	8	BITMAP64	0x1F	64-bit bitmap	uint8_t *
int8u	1	INT8U	0x20	Unsigned 8-bit integer	uint8_t
int16u	2	INT16U	0x21	Unsigned 16-bit integer	uint16_t
int24u	3	INT24U	0x22	Unsigned 24-bit integer	uint32_t
int32u	4	INT32U	0x23	Unsigned 32-bit integer	uint32_t
int40u	5	INT40U	0x24	Unsigned 40-bit integer	uint8_t *
int48u	6	INT48U	0x25	Unsigned 48-bit integer	uint8_t *
int56u	7	INT56U	0x26	Unsigned 56-bit integer	uint8_t *
int64u	8	INT64U	0x27	Unsigned 64-bit integer	uint8_t *
int8s	1	INT8S	0x28	Signed 8-bit integer	int8_t
int16s	2	INT16S	0x29	Signed 16-bit integer	int16_t
int24s	3	INT24S	0x2A	Signed 24-bit integer	int32_t
int32s	4	INT32S	0x2B	Signed 32-bit integer	int32_t
int40s	5	INT40S	0x2C	Signed 40-bit integer	int8_t *
int48s	6	INT48S	0x2D	Signed 48-bit integer	int8_t *
int56s	7	INT56S	0x2E	Signed 56-bit integer	int8_t *
int64s	8	INT64S	0x2F	Signed 64-bit integer	int8_t *
enum8	1	ENUM8	0x30	8-bit enumeration	uint8_t
enum16	2	ENUM16	0x31	16-bit enumeration	uint16_t
single	4	SINGLE	0x39	Single precision	uint8_t *
double	8	DOUBLE	0x3A	Double precision	uint8_t *
octet_string		OCTET_STRING	0x41	Octet String	uint8_t *
char_string		CHAR_STRING	0x42	Character String	uint8_t *
long_octet_string		LONG_OCTET_STRING	0x43	Long Octet String	uint8_t *
long_char_string		LONG_CHAR_STRING	0x44	Long Character String	uint8_t *
array		ARRAY	0x48	List	uint8_t *
struct		STRUCT	0x4C	Structure	uint8_t *
tod	4	TOD	0xE0	Time of day	uint8_t *
date	4	DATE	0xE1	Date	uint32_t
utc	4	UTC	0xE2	UTC Time	uint8_t *
epoch_us	8	EPOCH_US	0xE3	Epoch Microseconds	uint8_t *
epoch_s	4	EPOCH_S	0xE4	Epoch Seconds	uint8_t *
systime_us	8	SYSTIME_US	0xE5	System Time Microseconds	uint8_t *
percent	1	PERCENT	0xE6	Percentage units 1%	uint8_t *
percent100ths	2	PERCENT100THS	0xE7	Percentage units 0.01%	uint8_t *
cluster_id	4	CLUSTER_ID	0xE8	Cluster ID	uint16_t
attrib_id	4	ATTRIB_ID	0xE9	Attribute ID	uint8_t *
field_id	4	FIELD_ID	0xEA	Field ID	uint8_t *
event_id	4	EVENT_ID	0xEB	Event ID	uint8_t *
command_id	4	COMMAND_ID	0xEC	Command ID	uint8_t *
action_id	1	ACTION_ID	0xED	Action ID	uint8_t *
trans_id	4	TRANS_ID	0xEF	Transaction ID	uint8_t *
node_id	8	NODE_ID	0xF0	Node ID	uint8_t *
vendor_id	2	VENDOR_ID	0xF1	Vendor ID	uint8_t *
devtype_id	4	DEVTYPE_ID	0xF2	Device Type ID	uint8_t *
fabric_id	8	FABRIC_ID	0xF3	Fabric ID	uint8_t *
group_id	2	GROUP_ID	0xF4	Group ID	uint8_t *
status	2	STATUS	0xF5	Status Code	uint8_t *
data_ver	4	DATA_VER	0xF6	Data Version	uint_ver_t
event_no	8	EVENT_NO	0xF7	Event Number	uint8_t *
endpoint_no	2	ENDPOINT_NO	0xF8	Endpoint Number	uint8_t *
fabric_idx	1	FABRIC_IDX	0xF9	Fabric Index	uint8_t *
ipadr		IPADR	0xFA	IP Address	uint8_t *
ipv4adr	4	IPV4ADR	0xFB	IPv4 Address	uint8_t *
ipv6adr	16	IPV6ADR	0xFC	IPv6 Address	uint8_t *
ipv6pre		IPV6PRE	0xFD	IPv6 Prefix	uint8_t *
hwadr		HWADR	0xFE	Hardware Address	uint8_t *
unknown	0	UNKNOWN	0xFF	Unknown	uint8_t *

PowerConfiguration	AdministratorCommissioning	DissolvedOxygenConcentrationMeasurement
DeviceTemperatureConfiguration	OperationalCredentials	BromateConcentrationMeasurement
Identify	GroupKeyManagement	ChloraminesConcentrationMeasurement
Groups	FixedLabel	ChlorineConcentrationMeasurement
Scenes	UserLabel	FecalColiformAndEColiConcentrationMeasurement
OnOff	ProxyConfiguration	FluorideConcentrationMeasurement
OnOffSwitchConfiguration	ProxyDiscovery	HaloaceticAcidsConcentrationMeasurement
LevelControl	ProxyValid	TotalTrihalomethanesConcentrationMeasurement
Alarms	BooleanState	TotalColiformBacteriaConcentrationMeasurement
Time	ModeSelect	TurbidityConcentrationMeasurement
BinaryInputBasic	ShadeConfiguration	CopperConcentrationMeasurement
PowerProfile	DoorLock	LeadConcentrationMeasurement
ApplianceControl	WindowCovering	ManganeseConcentrationMeasurement
PulseWidthModulation	BarrierControl	SulfateConcentrationMeasurement
Descriptor	PumpConfigurationAndControl	BromodichloromethaneConcentrationMeasurement
Binding	Thermostat	BromoformConcentrationMeasurement
AccessControl	FanControl	ChlorodibromomethaneConcentrationMeasurement
PollControl	DehumidificationControl	ChloroformConcentrationMeasurement
Actions	ThermostatUserInterfaceConfiguration	SodiumConcentrationMeasurement
Basic	ColorControl	IasZone
OtaSoftwareUpdateProvider	BallastConfiguration	IasAce
OtaSoftwareUpdateRequestor	IlluminanceMeasurement	IasWd
LocalizationConfiguration	TemperatureMeasurement	WakeOnLan
TimeFormatLocalization	PressureMeasurement	Channel
UnitLocalization	FlowMeasurement	TargetNavigator
PowerSourceConfiguration	RelativeHumidityMeasurement	MediaPlayback
PowerSource	OccupancySensing	MediaInput
GeneralCommissioning	CarbonMonoxideConcentrationMeasurement	LowPower
NetworkCommissioning	CarbonDioxideConcentrationMeasurement	KeypadInput
DiagnosticLogs	EthyleneConcentrationMeasurement	ContentLauncher
GeneralDiagnostics	EthyleneOxideConcentrationMeasurement	AudioOutput
SoftwareDiagnostics	HydrogenConcentrationMeasurement	ApplicationLauncher
ThreadNetworkDiagnostics	HydrogenSulphideConcentrationMeasurement	ApplicationBasic
WiFiNetworkDiagnostics	NitricOxideConcentrationMeasurement	AccountLogin
EthernetNetworkDiagnostics	NitrogenDioxideConcentrationMeasurement	TestCluster
TimeSynchronization	OxygenConcentrationMeasurement	Messaging
BridgedDeviceBasic	OzoneConcentrationMeasurement	ApplianceIdentification
Switch	SulfurDioxideConcentrationMeasurement	MeterIdentification