Decode TLV payload data and present it in a human readable format (#27638)

* Start with a flat tree library for a human TLV format

* Temp change for test

* Switch to a flat list and more flexible finding ... expect I want to find by id AND name eventually

* Clang-format

* Add tests for searching by name in a flat tree

* Provide non-array find-entry

* Have a good tree position that works for navigating and descend/ascend

* Added more documentation

* Add more unit tests

* Fix naming

* Support current path for flat tree positions

* Restyle

* Add IM message encoding, to have pretty-print of data once available

* Added secure channel message formats

* Add UDC defintions

* Rename things

* make the matter file parseable

* Attempt to start a codegen for tlv meta mapping

* Restyle

* Add missing files

* Restyle

* Have some codegen working, start defining types and names

* Start implementing a bit of a table generation. not done, but tables start to exist

* Support events for tables

* Add support for commands (untested though)

* More work, all except lists and constants are code-generated

* Restyle

* List support and better tag support including anonymous support

* Make tags specific: many tags are NOT context tags currently

* Restyle

* Add some test data for development tests

* Start adding some test support ... to be removed later

* Code compiles

* Add a unit test that compiles and runs

* Starting some decoding support. Still very much broken

* A bit more decoding, this time we handle lists. TLV interface is VERY bad

* Better decoding, we now show data

* Add some item information, to prepare for enum and bitmap decoding

* Restyle

* Add error messages on usage of command line

* remove a non yes/no argument

* Update error syntax for 2 more arguments

* update the help. using both true/false and yes/no is a mess

* Update logic for decoding

* Do not allow restyle

* Better StringBuilder formatting

* Test adjustment

* Naming update

* Restyle

* Add Format option for buffer writers and string builders

* Update comments

* Updated to only have printf inside stringbuilder and NOT bufferwriter

* Restyle

* Add missing file

* remove cpp file comments

* Fix cast to make clang happy

* Much better formatting and make the compile clang-friendly

* minor const correctness change. TLVReader has non-const getters

* Add special tags for payloads of things

* Added logic for binary data and payloads, to process things

* Start adding clusters metadata, make everything const-correct

* Minor update

* Start updating formats

* Never pass null pointer in vsnprintf, since our size available is never 0

* Restyle

* Restyle

* Better decoding

* Clean up some printfs

* Iterator decoding seems to work, including getting sub-data types

* Better organization of code ... protocols decoder is actually a class now

* Allow passing in decode trees for protocol understanding

* Better arg parsing - was able to test for invalid data

* Restyle

* Add back reset call to stringbuilder

* Restyle and make protocol decoding actually work

* Unformatted protocols/cluster meta

* Add more trace data for testing, fix SEGFAULT in decoder

* Support non-struct list entries

* Switch list decoding logic to be inside generated metadata

* Restyle

* Fix compilation and generation

* Start having codegen support for protocols metadata

* Move clusters meta to compile time codegen as well

* Restyle

* Cleanup dependencies a bit

* Start making TestDecoding be actual unit tests

* more unit tests, without protocol decoding

* Slightly better formatting

* More unit tests ... although invalid data looks odd

* Better formatting of unknown attributes

* Updated tests

* Restyle

* Better messaging, test overflows

* Removed unused file

* Undo submodule update

* Add some tests for invoke. Command list is NOT complete

* Restyle

* Yield commands that have no request structure

* Fix comment

* Start adding some unit tests for cpp-tlvmeta codegen

* Add tests for real

* Allow both hex and json at the same time for output

* Rename log_json to just json

* Add file output option for json tracing

* make the output look like a json array when outputing to file

* Restyle

* Fix support of "json:log"

* Fix support of "json:log"

* make things compile

* Rename open/close to openfile/closefile to avoid override errors

* Restyle

* StartsWith should be available now globally as it is always used

* StartsWith should be available now globally as it is always used

* Forward declare json to make arm cross compile pass

* Forward declare json to make arm cross compile pass

* Restyle

* Add some support for formatting enums and bitmaps

* Restyle

* Add json_tracing exceptions for includes checks

* Proper bitmap support with tests, status codes are bitmaps now

* Restyle

* Update test for overflow to have more unique values

* Restyle

* Fix decoding of command inputs and names

* Add a fuzz test for payload decoder

* Handle invalid TLV

* Add more error handling. Fuzzing runs longer now

* Restyle

* Make clang happy

* Fix efr32 unit test compilation

* Fix python lint

* Restyle

* Fix typo and restyle

* Restyle

* Add dependencies to flat-tree for generated code: they are needed

* make tests use uppercase for unknown tags as this is the code update I made recently

* Undo submodule update

* Make clang-tidy happy

* Fix subscribe response message indexing to match spec

---------

Co-authored-by: Andrei Litvin <andreilitvin@google.com>
38 files changed
tree: f73871da4f8adcfa252eef3d0b5cdaac0a0b20a7
  1. .devcontainer/
  2. .githooks/
  3. .github/
  4. .vscode/
  5. build/
  6. build_overrides/
  7. config/
  8. credentials/
  9. docs/
  10. examples/
  11. integrations/
  12. scripts/
  13. src/
  14. third_party/
  15. zzz_generated/
  16. .clang-format
  17. .clang-tidy
  18. .default-version.min
  19. .dir-locals.el
  20. .editorconfig
  21. .flake8
  22. .gitattributes
  23. .gitignore
  24. .gitmodules
  25. .gn
  26. .isort.cfg
  27. .mergify.yml
  28. .prettierrc.json
  29. .pullapprove.yml
  30. .restyled.yaml
  31. .shellcheck_tree
  32. .spellcheck.yml
  33. BUILD.gn
  34. CODE_OF_CONDUCT.md
  35. CONTRIBUTING.md
  36. gn_build.sh
  37. kotlin-detect-config.yaml
  38. lgtm.yml
  39. LICENSE
  40. NOTICE
  41. README.md
  42. REVIEWERS.md
README.md

Matter

Builds

Builds

Android Ameba ASR BouffaloLab Darwin TI CC26X2X7 TI CC32XX EFR32 ESP32 Infineon i.MX Linux K32W with SE051 Linux ARM Linux Standalone Linux Standalone Mbed OS MW320 nRF Connect SDK Open IoT SDK QPG Telink Tizen

Tests

Unit / Integration Tests Cirque QEMU

Tools

ZAP Templates

Documentation

Documentation Build

About

Matter (formerly Project CHIP) creates more connections between more objects, simplifying development for manufacturers and increasing compatibility for consumers, guided by the Connectivity Standards Alliance.

What is Matter?

Matter is a unified, open-source application-layer connectivity standard built to enable developers and device manufacturers to connect and build reliable, and secure ecosystems and increase compatibility among connected home devices. It is built with market-proven technologies using Internet Protocol (IP) and is compatible with Thread and Wi-Fi network transports. Matter was developed by a Working Group within the Connectivity Standards Alliance (Alliance). This Working Group develops and promotes the adoption of the Matter standard, a royalty-free connectivity standard to increase compatibility among smart home products, with security as a fundamental design tenet. The vision that led major industry players to come together to build Matter is that smart connectivity should be simple, reliable, and interoperable.

Matter simplifies development for manufacturers and increases compatibility for consumers.

The standard was built around a shared belief that smart home devices should be secure, reliable, and seamless to use. By building upon Internet Protocol (IP), Matter enables communication across smart home devices, mobile apps, and cloud services and defines a specific set of IP-based networking technologies for device certification.

The Matter specification details everything necessary to implement a Matter application and transport layer stack. It is intended to be used by implementers as a complete specification.

The Alliance officially opened the Matter Working Group on January 17, 2020, and the specification is available for adoption now.

Visit buildwithmatter.com to learn more and read the latest news and updates about the project.

Project Overview

Development Goals

Matter is developed with the following goals and principles in mind:

Unifying: Matter is built with and on top of market-tested, existing technologies.

Interoperable: The specification permits communication between any Matter-certified device, subject to users’ permission.

Secure: The specification leverages modern security practices and protocols.

User Control: The end user controls authorization for interaction with devices.

Federated: No single entity serves as a throttle or a single point of failure for root of trust.

Robust: The set of protocols specifies a complete lifecycle of a device — starting with the seamless out-of-box experience, through operational protocols, to device and system management specifications required for proper function in the presence of change.

Low Overhead: The protocols are practically implementable on low compute-resource devices, such as MCUs.

Pervasive: The protocols are broadly deployable and accessible, by leveraging IP and being implementable on low-capability devices.

Ecosystem-Flexible: The protocol is flexible enough to accommodate deployment in ecosystems with differing policies.

Easy to Use: The protocol provides smooth, cohesive, integrated provisioning and out-of-box experience.

Open: The Project’s design and technical processes are open and transparent to the general public, including non-members wherever possible.

Architecture Overview

Matter aims to build a universal IPv6-based communication protocol for smart home devices. The protocol defines the application layer that will be deployed on devices and the different link layers to help maintain interoperability. The following diagram illustrates the normal operational mode of the stack: Matter Architecture Overview

The architecture is divided into layers to help separate the different responsibilities and introduce a good level of encapsulation among the various pieces of the protocol stack. The vast majority of interactions flow through the stack captured in the following Figure:

Matter Stack Architecture

  1. Application: High-order business logic of a device. For example, an application that is focused on lighting might contain logic to handle turning on/off the bulb as well as its color characteristics.
  1. Data Model: The data layer corresponds to the data and verb elements that help support the functionality of the application. The Application operates on these data structures when there is an intent to interact with the device.
  1. Interaction Model: The Interaction Model layer defines a set of interactions that can be performed between a client and server device. For example, reading or writing attributes on a server device would correspond to application behavior on the device. These interactions operate on the elements defined at the data model layer.
  1. Action Framing: Once an action is constructed using the Interaction Model, it is serialized into a prescribed packed binary format to encode for network transmission.
  1. Security: An encoded action frame is then sent down to the Security Layer to encrypt and sign the payload to ensure that data is secured and authenticated by both sender and receiver of a packet.

  2. Message Framing & Routing: With an interaction encrypted and signed, the Message Layer constructs the payload format with required and optional header fields; which specify the message's properties and some routing information.

  1. IP Framing & Transport Management: After the final payload has been constructed, it is sent to the underlying transport protocol for IP management of the data.

Current Status of Matter

Matter’s design and technical processes are intended to be open and transparent to the general public, including to Working Group non-members wherever possible. The availability of this GitHub repository and its source code under an Apache v2 license is an important and demonstrable step to achieving this commitment. Matter endeavors to bring together the best aspects of market-tested technologies and redeploy them as a unified and cohesive whole-system solution. The overall goal of this approach is to bring the benefits of Matter to consumers and manufacturers as quickly as possible. As a result, what you observe in this repository is an implementation-first approach to the technical specification, vetting integrations in practice. The Matter repository is growing and evolving to implement the overall architecture. The repository currently contains the security foundations, message framing and dispatch, and an implementation of the interaction model and data model. The code examples show simple interactions, and are supported on multiple transports -- Wi-Fi and Thread -- starting with resource-constrained (i.e., memory, processing) silicon platforms to help ensure Matter’s scalability.

How to Contribute

We welcome your contributions to Matter. Read our contribution guidelines here.

Building and Developing in Matter

Instructions about how to build Matter can be found here .

Directory Structure

The Matter repository is structured as follows:

File/FolderContent
buildBuild system support content and built output directories
build_overridesBuild system parameter customization for different platforms
configProject configurations
credentialsDevelopment and test credentials
docsDocumentation, including guides. Visit the Matter SDK documentation page to read it.
examplesExample firmware applications that demonstrate use of Matter
integrations3rd Party integrations
scriptsScripts needed to work with the Matter repository
srcImplementation of Matter
third_party3rd party code used by Matter
zzz_generatedzap generated template code - Revolving around cluster information
BUILD.gnBuild file for the gn build system
CODE_OF_CONDUCT.mdCode of conduct for Matter and contribution to it
CONTRIBUTING.mdGuidelines for contributing to Matter
LICENSEMatter license file
REVIEWERS.mdPR reviewers
gn_build.shBuild script for specific projects such as Android, EFR32, etc.
README.mdThis File

License

Matter is released under the Apache 2.0 license.