Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 1 | # Code generation |
| 2 | |
| 3 | ## Code generation inputs (`*.zap` files) |
| 4 | |
| 5 | Matter code relies on code generation for cluster-specific data types and |
| 6 | callbacks. Generally this is split into: |
| 7 | |
| 8 | - Data serialization for structures/lists/commands. This applies to both |
| 9 | client-side and server-side structures and objects |
| 10 | - Callback setup using the Ember-based framework. This generally applies to |
| 11 | server-side processing and the code generation defines what processing needs |
| 12 | to be done when a specific command is received or an attribute is read and |
| 13 | what memory should be allocated for storing cluster attributes |
| 14 | |
| 15 | Code generation depends on the clusters that are needed by an application. Every |
| 16 | application configures the specific set of endpoints and clusters it needs based |
| 17 | on the device type it supports. The selection of the supported clusters and |
| 18 | attributes (as optional attributes may be omitted to save memory) is generally |
| 19 | stored in `*.zap` files. |
| 20 | |
| 21 | The selection of enabled clusters and files is done using |
| 22 | [ZAP](https://github.com/project-chip/zap). You can download a recent release of |
| 23 | zap from its [releases page](https://github.com/project-chip/zap/releases). It |
| 24 | is recommended to download a release that is in sync with the currently in use |
Arkadiusz Bokowy | 3d9bc88 | 2023-02-14 03:01:17 +0100 | [diff] [blame] | 25 | version by the SDK (see `scripts/setup/zap.json` and |
Andrei Litvin | 73a188a | 2023-01-31 12:40:20 -0500 | [diff] [blame] | 26 | `scripts/tools/zap/zap_execution.py` for the minimum supported version). |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 27 | |
| 28 | Beyond basic zap file selection, there are also `.json` zap settings that define |
| 29 | additional cluster info: source XML files, sdk-access methods and data types. |
| 30 | There are only two such files currently in use: |
| 31 | |
| 32 | - `src/app/zap-templates/zcl/zcl.json` is the **default** one |
| 33 | - `src/app/zap-templates/zcl/zcl-with-test-extensions.json` is used by |
| 34 | `all-clusters-app` to show how a cluster extension may be configured with |
| 35 | minimal changes from `zcl.json` (but it is different) |
| 36 | |
| 37 | ### Installing zap and environment variables |
| 38 | |
Andrei Litvin | 73a188a | 2023-01-31 12:40:20 -0500 | [diff] [blame] | 39 | ZAP is generally installed as a third-party tool via CIPD during the build |
Arkadiusz Bokowy | 3d9bc88 | 2023-02-14 03:01:17 +0100 | [diff] [blame] | 40 | environment bootstrap (see `scripts/setup/zap.json`), which makes `zap-cli` |
| 41 | available in `$PATH` when running in a build environment. |
Andrei Litvin | 73a188a | 2023-01-31 12:40:20 -0500 | [diff] [blame] | 42 | |
Andrei Litvin | 73a188a | 2023-01-31 12:40:20 -0500 | [diff] [blame] | 43 | When matter scripts need to invoke `zap-cli` (for code generation) or `zap` (to |
| 44 | start the UI tool), they make use of the following environment variables to |
| 45 | figure out where the zap tool is located (in order of precedence): |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 46 | |
| 47 | - if `$ZAP_DEVELOPMENT_PATH` is set, code assumes you are running zap from |
| 48 | source. Use this if you develop zap. Zap has to be bootstrapped (generally |
| 49 | `npm ci` but check zap documentation for this. Some scripts have a |
| 50 | `--run-bootstrap` command line argument to do this for you) |
| 51 | |
| 52 | - if `$ZAP_INSTALL_PATH` is set, code assumes that `zap` or `zap-cli` is |
| 53 | available in the given path. This is generally an unpacked release. |
| 54 | |
Andrei Litvin | 73a188a | 2023-01-31 12:40:20 -0500 | [diff] [blame] | 55 | - otherwise, scripts will assume `zap`/`zap-cli` is in `$PATH` (this is the |
| 56 | case when running in a bootstrapped environment) |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 57 | |
| 58 | ### Using a UI to edit `.zap` files |
| 59 | |
| 60 | Generally you need to invoke zap with appropriate zcl and generate arguments. |
| 61 | Most of code generation is app specific, so you generally want something of the |
| 62 | form |
| 63 | `--gen src/app/zap-templates/app-templates.json --zcl $ZCL_JSON_FILE $ZAP_FILE_TO_EDIT` |
| 64 | |
| 65 | Since this is tedious to type, the SDK provides a |
| 66 | `scripts/tools/zap/run_zaptool.sh` script to automate this: |
| 67 | |
| 68 | ```bash |
Grant Erickson | a068855 | 2024-09-13 15:48:49 -0700 | [diff] [blame] | 69 | # Ensure `zap` is in $PATH, specify the `--zap ZAP` option to `run_zaptool.sh` to specify the path to `zap`, set $ZAP_INSTALL_PATH, or set $ZAP_DEVELOPMENT_PATH |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 70 | ./scripts/tools/zap/run_zaptool.sh examples/lighting-app/lighting-common/lighting-app.zap |
| 71 | ``` |
| 72 | |
| 73 | ### Human-readable code generation inputs (`*.matter`) |
| 74 | |
| 75 | `.zap` files are large json files that are generally not human readable. As a |
| 76 | result, the Matter SDK also keeps an equivalent `*.matter` file along side |
| 77 | `.zap` files that contain the same data as `.zap` files, targeted specifically |
| 78 | for matter: |
| 79 | |
| 80 | - They are designed to be human readable, looking like a IDL (think protobuf |
| 81 | or android `aidl`, thrift idl etc.) |
| 82 | |
| 83 | - We strive to make them contain only Matter-specific data (`.zap` files |
| 84 | contain more generic data and is designed to be ZigBee backwards compatible) |
| 85 | |
| 86 | Currently `.matter` files are generated from `.zap` files during the application |
| 87 | specific codegen. |
| 88 | |
| 89 | ### `*.matter` parsing and codegen |
| 90 | |
| 91 | `*.matter` files are both human and machine readable. Code that can process |
Terence Hampson | 260307b | 2023-01-05 09:56:23 -0500 | [diff] [blame] | 92 | these files is available at `scripts/py_matter_idl` and `scripts/codegen.py`. |
| 93 | You can read the |
| 94 | [scripts/py_matter_idl/matter_idl/README.md](../scripts/py_matter_idl/matter_idl/README.md) |
| 95 | for details of how things work. |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 96 | |
| 97 | `scripts/codegen.py` can generate various outputs based on an input `*.matter` |
| 98 | file. |
| 99 | |
| 100 | The split between `.zap` and `.matter` currently exists as an experiment of code |
| 101 | generation technologies. Currently `.matter`-based Python code generation: |
| 102 | |
| 103 | - has fewer third party dependencies than `zap`, which installs a significant |
| 104 | number of `npm` packages. |
| 105 | - runs significantly faster than zap |
| 106 | - offers more flexible code generation (can generate multiple files per |
| 107 | cluster for example, without which some compiles would run out of RAM on |
| 108 | large compilations) |
| 109 | - has a more flexible templating language |
| 110 | - has human readable (and potentially editable) input |
| 111 | - is more easily provable deterministic (`zap` uses an underlying sqlite |
| 112 | database and some legacy assumptions from zigbee have historically caused |
| 113 | non-determinism) |
| 114 | - uses a synchronous processing model which is potentially easier to develop |
| 115 | for |
| 116 | - has lower complexity, is unit tested and uses typing extensively |
| 117 | |
| 118 | Ideally, the project would be to have a single code generation method in the |
| 119 | long term that has all the benefits and none of the drawbacks. We are not there |
| 120 | yet, however we likely want: |
| 121 | |
| 122 | - Flexible codegen (we will need to split output by clusters or other rules) |
| 123 | - Human-readable inputs that enable code reviews and audits |
| 124 | - Rules that a script can validate based on CSA data model (ensure mandatory |
| 125 | attribute settings are followed, ensure proper device type adherence, ensure |
| 126 | correct cluster and data type definitions) |
| 127 | - Easy to maintain and develop for chosen languages/templates/codegen in |
| 128 | general |
| 129 | |
| 130 | ## Code generation outputs and templates |
| 131 | |
| 132 | Code that is generated: |
| 133 | |
| 134 | - **Application-specific**: |
| 135 | |
| 136 | - ZAP generation is based on `.zap` files in `examples/` and generates |
| 137 | server-side processing data: what cluster callbacks to set up, what RAM |
| 138 | to reserve for attribute storage etc. |
| 139 | |
| 140 | - `Codegen.py` will also generate a subset of application-specific files |
| 141 | |
| 142 | - **Automated tests**: embedded client-side tools (`chip-tool` and |
| 143 | `darwin-framework-tool`) generate test-definition data. Each use their own |
| 144 | `examples/${TOOL}/templates/tests/templates.json` to drive what gets |
| 145 | generated. |
| 146 | |
| 147 | - **Controller clusters** target: the file |
| 148 | `src/controller/data_model/controller-clusters.zap` contains a set of |
| 149 | cluster selections to which all applications would potentially have access. |
| 150 | These are generally used as `all clusters selection` and the intent is to |
| 151 | allow any application to access any cluster as a `client side`. |
| 152 | |
| 153 | Client/controllers will codegen based on this, like **tools**, **tests**, |
| 154 | **java**, **python** etc. |
| 155 | |
| 156 | ## Running codegen |
| 157 | |
| 158 | ### ZAP file generation |
| 159 | |
| 160 | Generating all possible code (all categories above) using zap tool can be done |
| 161 | via: |
| 162 | |
| 163 | ```bash |
| 164 | ./scripts/tools/zap_regen_all.py |
| 165 | ``` |
| 166 | |
| 167 | This can be slow (several minutes). The regen tool allows selection of only |
| 168 | tests so that yaml test development goes faster. |
| 169 | |
| 170 | ```bash |
| 171 | ./scripts/tools/zap_regen_all.py --type tests |
| 172 | ./scripts/tools/zap_regen_all.py --type tests --tests chip-tool |
| 173 | ``` |
| 174 | |
| 175 | Additionally, individual code regeneration can be done using |
| 176 | `./scripts/tools/zap/generate.py`: |
| 177 | |
| 178 | ```bash |
Andrei Litvin | 4d9df98 | 2022-11-30 11:12:43 -0500 | [diff] [blame] | 179 | /scripts/tools/zap/generate.py \ |
Andrei Litvin | 3477068 | 2023-01-31 11:39:22 -0500 | [diff] [blame] | 180 | examples/bridge-app/bridge-common/bridge-app.zap |
| 181 | ``` |
| 182 | |
| 183 | The above will just generate a `<app>.matter` file along side the `.zap` file, |
| 184 | as this is the only file that requires updates for applications. You can code |
| 185 | generate other things by passing in the `-t/--templates` argument to |
| 186 | generate.py. In those cases, you may also need to specify an output directory |
| 187 | via `-o/--output-dir`. |
| 188 | |
| 189 | #### Flow for updating an application zap file: |
| 190 | |
| 191 | ``` |
| 192 | # use zap UI to edit the file (or edit zap file in any other way) |
| 193 | ./scripts/tools/zap/run_zaptool.sh $PATH_TO_ZAP_FILE |
| 194 | |
| 195 | # re-generate .matter file. Note that for .matter file generation, output |
| 196 | # directory is NOT used |
| 197 | ./scripts/tools/zap/generate.py $PATH_TO_ZAP_FILE |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 198 | ``` |
| 199 | |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 200 | ### Compile-time code generation / pre-generated code |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 201 | |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 202 | A subset of code generation (both `codegen.py` and `zap-cli`) is done at compile |
| 203 | time or can use pre-generated output (based on gn/cmake arguments) |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 204 | |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 205 | Rules for how `generate.py`/`codegen.py` is invoked at compile time are defined |
| 206 | at: |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 207 | |
| 208 | - `src/app/chip_data_model.cmake` |
Andrei Litvin | 6026c60 | 2022-11-24 10:01:02 -0500 | [diff] [blame] | 209 | - `src/app/chip_data_model.gni` |
Andrei Litvin | a715884 | 2022-11-30 10:11:41 -0500 | [diff] [blame] | 210 | |
| 211 | Additionally, `build/chip/esp32/esp32_codegen.cmake` adds processing support for |
| 212 | the 2-pass cmake builds used by the Espressif `idf.py` build system. |
| 213 | |
| 214 | ## Pre-generation |
| 215 | |
| 216 | Code pre-generation can be used: |
| 217 | |
| 218 | - when compile-time code generation is not desirable. This may be for |
| 219 | importing into build systems that do not have the pre-requisites to run code |
| 220 | generation at build time or to save the code generation time at the expense |
| 221 | of running code generation for every possible zap/generation type |
| 222 | - To check changes in generated code across versions, beyond the comparisons |
Terence Hampson | 260307b | 2023-01-05 09:56:23 -0500 | [diff] [blame] | 223 | of golden image tests in `scripts/py_matter_idl/matter_idl/tests` |
Andrei Litvin | a715884 | 2022-11-30 10:11:41 -0500 | [diff] [blame] | 224 | |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 225 | The script to trigger code pre-generation is `scripts/codepregen.py` and |
Andrei Litvin | a715884 | 2022-11-30 10:11:41 -0500 | [diff] [blame] | 226 | requires the pre-generation output directory as an argument |
| 227 | |
| 228 | ```bash |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 229 | scripts/codepregen.py ${OUTPUT_DIRECTORY:-./zzz_pregenerated/} |
| 230 | |
| 231 | # To generate a single output you can use `--input-glob`: |
| 232 | |
Andrei Litvin | ef05822 | 2023-04-11 13:30:08 -0400 | [diff] [blame] | 233 | scripts/codepregen.py --input-glob "*all-clusters*" --input-glob "*controller*" ${OUTPUT_DIRECTORY:-./zzz_pregenerated/} |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 234 | ``` |
| 235 | |
Andrei Litvin | ef05822 | 2023-04-11 13:30:08 -0400 | [diff] [blame] | 236 | ### External applications/zap files |
| 237 | |
| 238 | #### Ensure you have a `.matter` file |
| 239 | |
| 240 | Code generation generally will use both `.zap` or `.matter` files. If you only |
| 241 | have a `.zap` file, you can create the corresponding `.matter` file via: |
| 242 | |
| 243 | ```bash |
| 244 | scripts/tools/zap/generate.py ${ZAP_FILE_PATH} |
| 245 | ``` |
| 246 | |
| 247 | The above will use the template `src/app/zap-templates/matter-idl.json` to |
| 248 | generate a `.matter` file corresponding to the input `.zap` file. |
| 249 | |
| 250 | `.matter` files are designed to be human readable. It is recommended to take a |
| 251 | look at the generated file and see if it contains what is expected and also lint |
| 252 | it. If anything seems wrong, the `.zap` file should be fixed (`.matter` |
| 253 | represents the content of `.zap`). To lint use: |
| 254 | |
| 255 | ```bash |
| 256 | scripts/idl_lint.py ${MATTER_FILE_PATH} |
| 257 | ``` |
| 258 | |
| 259 | #### Running pre-generation |
| 260 | |
| 261 | If you have zap files outside the CHIP repository (i.e. not in `src` or |
| 262 | `examples`) you should provide the root of your application source. |
| 263 | |
| 264 | ```bash |
| 265 | scripts/codepregen.py --external-root ${PATH_TO_SOURCE_ROOT} ${OUTPUT_DIRECTORY:-./zzz_pregenerated/} |
| 266 | ``` |
| 267 | |
| 268 | NOTE: `$PATH_TO_SOURCE_ROOT` should be a top-level directory containing |
| 269 | zap/matter files as the code pre-generation will generate files based on the |
| 270 | path inside the root: |
| 271 | |
| 272 | - if files are `$PATH_TO_SOURCE_ROOT/some/path/foo.zap` this will generate |
| 273 | files into `$OUTPUT_DIRECTORY/some/path/foo/...` |
| 274 | |
Andrei Litvin | 89c6fb6 | 2023-01-17 21:29:11 -0500 | [diff] [blame] | 275 | ### Using pre-generated code |
| 276 | |
| 277 | Instead of generating code at compile time, the chip build system accepts usage |
| 278 | of a pre-generated folder. It assumes the structure that `codepregen.py` |
| 279 | creates. To invoke use: |
| 280 | |
| 281 | - `build_examples.py` builds accept `--pregen-dir` as an argument, such as: |
| 282 | |
| 283 | ```shell |
| 284 | ./scripts/build/build_examples.py --target $TARGET --pregen-dir $PREGEN_DIR build |
| 285 | ``` |
| 286 | |
| 287 | - `gn` builds allow setting `chip_code_pre_generated_directory` as an |
| 288 | argument, such as: |
| 289 | |
| 290 | ```shell |
| 291 | gn gen --check --fail-on-unused-args --args='chip_code_pre_generated_directory="/some/pregen/dir"' |
| 292 | ``` |
| 293 | |
| 294 | - `cmake` builds allow setting `CHIP_CODEGEN_PREGEN_DIR` variable (which will |
| 295 | get propagated to the underlying `gn` builds as needed), such as: |
| 296 | |
| 297 | ```shell |
| 298 | |
| 299 | west build --cmake-only \ |
| 300 | -d /workspace/out/nrf-nrf5340dk-light \ |
| 301 | -b nrf5340dk_nrf5340_cpuapp \ |
| 302 | /workspace/examples/lighting-app/nrfconnect |
| 303 | -- -DCHIP_CODEGEN_PREGEN_DIR=/some/pregen/dir |
| 304 | |
| 305 | idf.py -C examples/all-clusters-app/esp32 \ |
| 306 | -B /workspace/out/esp32-m5stack-all-clusters \ |
| 307 | -DCHIP_CODEGEN_PREGEN_DIR=/some/pregen/dir \ |
| 308 | reconfigure |
| 309 | |
| 310 | cmake -S /workspace/examples/lighting-app/mbed \ |
| 311 | -B /workspace/out/mbed-cy8cproto_062_4343w-light \ |
| 312 | -GNinja \ |
| 313 | -DMBED_OS_PATH=/workspace/third_party/mbed-os/repo \ |
| 314 | -DMBED_OS_PATH=/workspace/third_party/mbed-os/repo \ |
| 315 | -DMBED_OS_POSIX_SOCKET_PATH=/workspace/third_party/mbed-os-posix-socket/repo \ |
| 316 | -DCHIP_CODEGEN_PREGEN_DIR=/some/pregen/dir |
| 317 | ``` |
| 318 | |
| 319 | ### Code generation unit testing |
| 320 | |
| 321 | Code generation is assumed stable between builds and the build system aims to |
| 322 | detect changes in code gen using golden image tests. |
| 323 | |
| 324 | #### `codegen.py` tests |
| 325 | |
| 326 | These tests run against golden inputs/outputs from `scripts/idl/tests`. |
| 327 | |
| 328 | `available_tests.yaml` contains the full list of expected generators and outputs |
| 329 | and the test is run via `test_generators.py`. Use the environment variable |
| 330 | `IDL_GOLDEN_REGENERATE` to force golden image replacement during running of |
| 331 | `ninja check`: |
| 332 | |
| 333 | ```shell |
| 334 | IDL_GOLDEN_REGENERATE=1 ninja check |
| 335 | ``` |
| 336 | |
| 337 | #### `generate.py` tests |
| 338 | |
| 339 | These tests run against golden inputs/outputs from `scripts/tools/zap/tests`. |
| 340 | |
| 341 | `available_tests.yaml` contains the full list of expected generators and outputs |
| 342 | and the test is run via `scripts/tools/zap/test_generate.py`. Use the |
| 343 | environment variable `ZAP_GENERATE_GOLDEN_REGENERATE` to force golden image |
| 344 | replacement during running of `ninja check`. |
| 345 | |
| 346 | ```shell |
| 347 | ZAP_GENERATE_GOLDEN_REGENERATE=1 ninja check |
Andrei Litvin | a715884 | 2022-11-30 10:11:41 -0500 | [diff] [blame] | 348 | ``` |
Andrei Litvin | b88fce0 | 2023-01-20 12:21:49 -0500 | [diff] [blame] | 349 | |
| 350 | Alternatively, the golden image can also be re-generated by running the |
| 351 | stand-alone test in a bootstrapped environment: |
| 352 | |
| 353 | ```shell |
| 354 | ./scripts/tools/zap/test_generate.py --output out/gen --regenerate |
| 355 | ``` |