blob: bf5b6f3036963a044462f8452865c53bb3b0c4f9 [file] [log] [blame] [view]
Andrei Litvin6026c602022-11-24 10:01:02 -05001# Code generation
2
3## Code generation inputs (`*.zap` files)
4
5Matter code relies on code generation for cluster-specific data types and
6callbacks. Generally this is split into:
7
8- Data serialization for structures/lists/commands. This applies to both
9 client-side and server-side structures and objects
10- Callback setup using the Ember-based framework. This generally applies to
11 server-side processing and the code generation defines what processing needs
12 to be done when a specific command is received or an attribute is read and
13 what memory should be allocated for storing cluster attributes
14
15Code generation depends on the clusters that are needed by an application. Every
16application configures the specific set of endpoints and clusters it needs based
17on the device type it supports. The selection of the supported clusters and
18attributes (as optional attributes may be omitted to save memory) is generally
19stored in `*.zap` files.
20
21The selection of enabled clusters and files is done using
22[ZAP](https://github.com/project-chip/zap). You can download a recent release of
23zap from its [releases page](https://github.com/project-chip/zap/releases). It
24is recommended to download a release that is in sync with the currently in use
Arkadiusz Bokowy3d9bc882023-02-14 03:01:17 +010025version by the SDK (see `scripts/setup/zap.json` and
Andrei Litvin73a188a2023-01-31 12:40:20 -050026`scripts/tools/zap/zap_execution.py` for the minimum supported version).
Andrei Litvin6026c602022-11-24 10:01:02 -050027
28Beyond basic zap file selection, there are also `.json` zap settings that define
29additional cluster info: source XML files, sdk-access methods and data types.
30There are only two such files currently in use:
31
32- `src/app/zap-templates/zcl/zcl.json` is the **default** one
33- `src/app/zap-templates/zcl/zcl-with-test-extensions.json` is used by
34 `all-clusters-app` to show how a cluster extension may be configured with
35 minimal changes from `zcl.json` (but it is different)
36
37### Installing zap and environment variables
38
Andrei Litvin73a188a2023-01-31 12:40:20 -050039ZAP is generally installed as a third-party tool via CIPD during the build
Arkadiusz Bokowy3d9bc882023-02-14 03:01:17 +010040environment bootstrap (see `scripts/setup/zap.json`), which makes `zap-cli`
41available in `$PATH` when running in a build environment.
Andrei Litvin73a188a2023-01-31 12:40:20 -050042
Andrei Litvin73a188a2023-01-31 12:40:20 -050043When matter scripts need to invoke `zap-cli` (for code generation) or `zap` (to
44start the UI tool), they make use of the following environment variables to
45figure out where the zap tool is located (in order of precedence):
Andrei Litvin6026c602022-11-24 10:01:02 -050046
47- if `$ZAP_DEVELOPMENT_PATH` is set, code assumes you are running zap from
48 source. Use this if you develop zap. Zap has to be bootstrapped (generally
49 `npm ci` but check zap documentation for this. Some scripts have a
50 `--run-bootstrap` command line argument to do this for you)
51
52- if `$ZAP_INSTALL_PATH` is set, code assumes that `zap` or `zap-cli` is
53 available in the given path. This is generally an unpacked release.
54
Andrei Litvin73a188a2023-01-31 12:40:20 -050055- otherwise, scripts will assume `zap`/`zap-cli` is in `$PATH` (this is the
56 case when running in a bootstrapped environment)
Andrei Litvin6026c602022-11-24 10:01:02 -050057
58### Using a UI to edit `.zap` files
59
60Generally you need to invoke zap with appropriate zcl and generate arguments.
61Most of code generation is app specific, so you generally want something of the
62form
63`--gen src/app/zap-templates/app-templates.json --zcl $ZCL_JSON_FILE $ZAP_FILE_TO_EDIT`
64
65Since this is tedious to type, the SDK provides a
66`scripts/tools/zap/run_zaptool.sh` script to automate this:
67
68```bash
Grant Ericksona0688552024-09-13 15:48:49 -070069# Ensure `zap` is in $PATH, specify the `--zap ZAP` option to `run_zaptool.sh` to specify the path to `zap`, set $ZAP_INSTALL_PATH, or set $ZAP_DEVELOPMENT_PATH
Andrei Litvin6026c602022-11-24 10:01:02 -050070./scripts/tools/zap/run_zaptool.sh examples/lighting-app/lighting-common/lighting-app.zap
71```
72
73### Human-readable code generation inputs (`*.matter`)
74
75`.zap` files are large json files that are generally not human readable. As a
76result, the Matter SDK also keeps an equivalent `*.matter` file along side
77`.zap` files that contain the same data as `.zap` files, targeted specifically
78for matter:
79
80- They are designed to be human readable, looking like a IDL (think protobuf
81 or android `aidl`, thrift idl etc.)
82
83- We strive to make them contain only Matter-specific data (`.zap` files
84 contain more generic data and is designed to be ZigBee backwards compatible)
85
86Currently `.matter` files are generated from `.zap` files during the application
87specific codegen.
88
89### `*.matter` parsing and codegen
90
91`*.matter` files are both human and machine readable. Code that can process
Terence Hampson260307b2023-01-05 09:56:23 -050092these files is available at `scripts/py_matter_idl` and `scripts/codegen.py`.
93You can read the
94[scripts/py_matter_idl/matter_idl/README.md](../scripts/py_matter_idl/matter_idl/README.md)
95for details of how things work.
Andrei Litvin6026c602022-11-24 10:01:02 -050096
97`scripts/codegen.py` can generate various outputs based on an input `*.matter`
98file.
99
100The split between `.zap` and `.matter` currently exists as an experiment of code
101generation technologies. Currently `.matter`-based Python code generation:
102
103- has fewer third party dependencies than `zap`, which installs a significant
104 number of `npm` packages.
105- runs significantly faster than zap
106- offers more flexible code generation (can generate multiple files per
107 cluster for example, without which some compiles would run out of RAM on
108 large compilations)
109- has a more flexible templating language
110- has human readable (and potentially editable) input
111- is more easily provable deterministic (`zap` uses an underlying sqlite
112 database and some legacy assumptions from zigbee have historically caused
113 non-determinism)
114- uses a synchronous processing model which is potentially easier to develop
115 for
116- has lower complexity, is unit tested and uses typing extensively
117
118Ideally, the project would be to have a single code generation method in the
119long term that has all the benefits and none of the drawbacks. We are not there
120yet, however we likely want:
121
122- Flexible codegen (we will need to split output by clusters or other rules)
123- Human-readable inputs that enable code reviews and audits
124- Rules that a script can validate based on CSA data model (ensure mandatory
125 attribute settings are followed, ensure proper device type adherence, ensure
126 correct cluster and data type definitions)
127- Easy to maintain and develop for chosen languages/templates/codegen in
128 general
129
130## Code generation outputs and templates
131
132Code that is generated:
133
134- **Application-specific**:
135
136 - ZAP generation is based on `.zap` files in `examples/` and generates
137 server-side processing data: what cluster callbacks to set up, what RAM
138 to reserve for attribute storage etc.
139
140 - `Codegen.py` will also generate a subset of application-specific files
141
142- **Automated tests**: embedded client-side tools (`chip-tool` and
143 `darwin-framework-tool`) generate test-definition data. Each use their own
144 `examples/${TOOL}/templates/tests/templates.json` to drive what gets
145 generated.
146
147- **Controller clusters** target: the file
148 `src/controller/data_model/controller-clusters.zap` contains a set of
149 cluster selections to which all applications would potentially have access.
150 These are generally used as `all clusters selection` and the intent is to
151 allow any application to access any cluster as a `client side`.
152
153 Client/controllers will codegen based on this, like **tools**, **tests**,
154 **java**, **python** etc.
155
156## Running codegen
157
158### ZAP file generation
159
160Generating all possible code (all categories above) using zap tool can be done
161via:
162
163```bash
164./scripts/tools/zap_regen_all.py
165```
166
167This can be slow (several minutes). The regen tool allows selection of only
168tests so that yaml test development goes faster.
169
170```bash
171./scripts/tools/zap_regen_all.py --type tests
172./scripts/tools/zap_regen_all.py --type tests --tests chip-tool
173```
174
175Additionally, individual code regeneration can be done using
176`./scripts/tools/zap/generate.py`:
177
178```bash
Andrei Litvin4d9df982022-11-30 11:12:43 -0500179/scripts/tools/zap/generate.py \
Andrei Litvin34770682023-01-31 11:39:22 -0500180 examples/bridge-app/bridge-common/bridge-app.zap
181```
182
183The above will just generate a `<app>.matter` file along side the `.zap` file,
184as this is the only file that requires updates for applications. You can code
185generate other things by passing in the `-t/--templates` argument to
186generate.py. In those cases, you may also need to specify an output directory
187via `-o/--output-dir`.
188
189#### Flow for updating an application zap file:
190
191```
192# use zap UI to edit the file (or edit zap file in any other way)
193./scripts/tools/zap/run_zaptool.sh $PATH_TO_ZAP_FILE
194
195# re-generate .matter file. Note that for .matter file generation, output
196# directory is NOT used
197./scripts/tools/zap/generate.py $PATH_TO_ZAP_FILE
Andrei Litvin6026c602022-11-24 10:01:02 -0500198```
199
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500200### Compile-time code generation / pre-generated code
Andrei Litvin6026c602022-11-24 10:01:02 -0500201
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500202A subset of code generation (both `codegen.py` and `zap-cli`) is done at compile
203time or can use pre-generated output (based on gn/cmake arguments)
Andrei Litvin6026c602022-11-24 10:01:02 -0500204
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500205Rules for how `generate.py`/`codegen.py` is invoked at compile time are defined
206at:
Andrei Litvin6026c602022-11-24 10:01:02 -0500207
208- `src/app/chip_data_model.cmake`
Andrei Litvin6026c602022-11-24 10:01:02 -0500209- `src/app/chip_data_model.gni`
Andrei Litvina7158842022-11-30 10:11:41 -0500210
211Additionally, `build/chip/esp32/esp32_codegen.cmake` adds processing support for
212the 2-pass cmake builds used by the Espressif `idf.py` build system.
213
214## Pre-generation
215
216Code pre-generation can be used:
217
218- when compile-time code generation is not desirable. This may be for
219 importing into build systems that do not have the pre-requisites to run code
220 generation at build time or to save the code generation time at the expense
221 of running code generation for every possible zap/generation type
222- To check changes in generated code across versions, beyond the comparisons
Terence Hampson260307b2023-01-05 09:56:23 -0500223 of golden image tests in `scripts/py_matter_idl/matter_idl/tests`
Andrei Litvina7158842022-11-30 10:11:41 -0500224
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500225The script to trigger code pre-generation is `scripts/codepregen.py` and
Andrei Litvina7158842022-11-30 10:11:41 -0500226requires the pre-generation output directory as an argument
227
228```bash
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500229scripts/codepregen.py ${OUTPUT_DIRECTORY:-./zzz_pregenerated/}
230
231# To generate a single output you can use `--input-glob`:
232
Andrei Litvinef058222023-04-11 13:30:08 -0400233scripts/codepregen.py --input-glob "*all-clusters*" --input-glob "*controller*" ${OUTPUT_DIRECTORY:-./zzz_pregenerated/}
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500234```
235
Andrei Litvinef058222023-04-11 13:30:08 -0400236### External applications/zap files
237
238#### Ensure you have a `.matter` file
239
240Code generation generally will use both `.zap` or `.matter` files. If you only
241have a `.zap` file, you can create the corresponding `.matter` file via:
242
243```bash
244scripts/tools/zap/generate.py ${ZAP_FILE_PATH}
245```
246
247The above will use the template `src/app/zap-templates/matter-idl.json` to
248generate a `.matter` file corresponding to the input `.zap` file.
249
250`.matter` files are designed to be human readable. It is recommended to take a
251look at the generated file and see if it contains what is expected and also lint
252it. If anything seems wrong, the `.zap` file should be fixed (`.matter`
253represents the content of `.zap`). To lint use:
254
255```bash
256scripts/idl_lint.py ${MATTER_FILE_PATH}
257```
258
259#### Running pre-generation
260
261If you have zap files outside the CHIP repository (i.e. not in `src` or
262`examples`) you should provide the root of your application source.
263
264```bash
265scripts/codepregen.py --external-root ${PATH_TO_SOURCE_ROOT} ${OUTPUT_DIRECTORY:-./zzz_pregenerated/}
266```
267
268NOTE: `$PATH_TO_SOURCE_ROOT` should be a top-level directory containing
269zap/matter files as the code pre-generation will generate files based on the
270path inside the root:
271
272- if files are `$PATH_TO_SOURCE_ROOT/some/path/foo.zap` this will generate
273 files into `$OUTPUT_DIRECTORY/some/path/foo/...`
274
Andrei Litvin89c6fb62023-01-17 21:29:11 -0500275### Using pre-generated code
276
277Instead of generating code at compile time, the chip build system accepts usage
278of a pre-generated folder. It assumes the structure that `codepregen.py`
279creates. To invoke use:
280
281- `build_examples.py` builds accept `--pregen-dir` as an argument, such as:
282
283 ```shell
284 ./scripts/build/build_examples.py --target $TARGET --pregen-dir $PREGEN_DIR build
285 ```
286
287- `gn` builds allow setting `chip_code_pre_generated_directory` as an
288 argument, such as:
289
290 ```shell
291 gn gen --check --fail-on-unused-args --args='chip_code_pre_generated_directory="/some/pregen/dir"'
292 ```
293
294- `cmake` builds allow setting `CHIP_CODEGEN_PREGEN_DIR` variable (which will
295 get propagated to the underlying `gn` builds as needed), such as:
296
297 ```shell
298
299 west build --cmake-only \
300 -d /workspace/out/nrf-nrf5340dk-light \
301 -b nrf5340dk_nrf5340_cpuapp \
302 /workspace/examples/lighting-app/nrfconnect
303 -- -DCHIP_CODEGEN_PREGEN_DIR=/some/pregen/dir
304
305 idf.py -C examples/all-clusters-app/esp32 \
306 -B /workspace/out/esp32-m5stack-all-clusters \
307 -DCHIP_CODEGEN_PREGEN_DIR=/some/pregen/dir \
308 reconfigure
309
310 cmake -S /workspace/examples/lighting-app/mbed \
311 -B /workspace/out/mbed-cy8cproto_062_4343w-light \
312 -GNinja \
313 -DMBED_OS_PATH=/workspace/third_party/mbed-os/repo \
314 -DMBED_OS_PATH=/workspace/third_party/mbed-os/repo \
315 -DMBED_OS_POSIX_SOCKET_PATH=/workspace/third_party/mbed-os-posix-socket/repo \
316 -DCHIP_CODEGEN_PREGEN_DIR=/some/pregen/dir
317 ```
318
319### Code generation unit testing
320
321Code generation is assumed stable between builds and the build system aims to
322detect changes in code gen using golden image tests.
323
324#### `codegen.py` tests
325
326These tests run against golden inputs/outputs from `scripts/idl/tests`.
327
328`available_tests.yaml` contains the full list of expected generators and outputs
329and the test is run via `test_generators.py`. Use the environment variable
330`IDL_GOLDEN_REGENERATE` to force golden image replacement during running of
331`ninja check`:
332
333```shell
334IDL_GOLDEN_REGENERATE=1 ninja check
335```
336
337#### `generate.py` tests
338
339These tests run against golden inputs/outputs from `scripts/tools/zap/tests`.
340
341`available_tests.yaml` contains the full list of expected generators and outputs
342and the test is run via `scripts/tools/zap/test_generate.py`. Use the
343environment variable `ZAP_GENERATE_GOLDEN_REGENERATE` to force golden image
344replacement during running of `ninja check`.
345
346```shell
347ZAP_GENERATE_GOLDEN_REGENERATE=1 ninja check
Andrei Litvina7158842022-11-30 10:11:41 -0500348```
Andrei Litvinb88fce02023-01-20 12:21:49 -0500349
350Alternatively, the golden image can also be re-generated by running the
351stand-alone test in a bootstrapped environment:
352
353```shell
354./scripts/tools/zap/test_generate.py --output out/gen --regenerate
355```