Fixed python test runner code to error out if there are 0 test runs (#34164)
* Create python_test_arguments.txt
* Fixed runner code that 0 runs are an error
* Restyled by autopep8
* Update hello_test.py
* Update python_test_arguments.txt
* Update python.md
* Update python.md
* Delete docs/testing/python_test_arguments.txt
* Restyled by prettier-markdown
* Update python.md
* Update python.md
* Update run_python_test.py
* Restyled by prettier-markdown
* Update python.md
* Restyled by prettier-markdown
---------
Co-authored-by: Restyled.io <commits@restyled.io>
diff --git a/docs/testing/python.md b/docs/testing/python.md
index 51f734a..06051fe 100644
--- a/docs/testing/python.md
+++ b/docs/testing/python.md
@@ -19,6 +19,11 @@
## Writing Python tests
+- Defining arguments in the test script
+ - In order to streamline the configuration and execution of tests, it is
+ essential to define arguments at the top of the test script. This
+ section should include various parameters and their respective values,
+ which will guide the test runner on how to execute the tests.
- All test classes inherit from MatterBaseTest in
[matter_testing_support.py](https://github.com/project-chip/connectedhomeip/blob/master/src/python_testing/matter_testing_support.py)
- support for commissioning using the python controller
@@ -36,11 +41,17 @@
- Use Mobly assertions for failing tests
- self.step() along with a steps\_ function to mark test plan steps for cert
tests
--
### A simple test
```
+# test-runner-runs: run1
+# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
+# test-runner-run/run1/factoryreset: True
+# test-runner-run/run1/quiet: True
+# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
+# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
class TC_MYTEST_1_1(MatterBaseTest):
@async_test_body
@@ -74,6 +85,59 @@
line. These two lines should appear verbatim at the bottom of every python test
file.
+## Defining the test arguments
+
+Below is the format:
+
+```
+# test-runner-runs: <run_identifier>
+# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
+# test-runner-run/<run_identifier>/factoryreset: <True|False>
+# test-runner-run/<run_identifier>/quiet: <True|False>
+# test-runner-run/<run_identifier>/app-args: <app_arguments>
+# test-runner-run/<run_identifier>/script-args: <script_arguments>
+```
+
+### Description of Parameters
+
+- test-runner-runs: Specifies the identifier for the run. This can be any
+ unique identifier.
+
+ - Example: run1
+
+- test-runner-run/<run_identifier>/app: Indicates the application to be used
+ in the test. Different app types as needed could be referenced from section
+ [name: Generate an argument environment file ] of the file
+ [.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)
+
+ - Example: \${TYPE_OF_APP}
+
+- test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
+ reset should be performed before the test.
+
+ - Example: True
+
+- test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
+ run. When set to True, the test run will be quieter.
+
+ - Example: True
+
+- test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
+ passed to the application during the test.
+
+ - Example: --discriminator 1234 --KVS kvs1 --trace-to
+ json:\${TRACE_APP}.json
+
+- test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
+ passed to the test script.
+ - Example: --storage-path admin_storage.json --commissioning-method
+ on-network --discriminator 1234 --passcode 20202021 --trace-to
+ json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
+This structured format ensures that all necessary configurations are clearly
+defined and easily understood, allowing for consistent and reliable test
+execution.
+
## Cluster Codegen
- [Objects.py](https://github.com/project-chip/connectedhomeip/blob/master/src/controller/python/chip/clusters/Objects.py)
diff --git a/scripts/tests/run_python_test.py b/scripts/tests/run_python_test.py
index 5ccd67a..7d7500c 100755
--- a/scripts/tests/run_python_test.py
+++ b/scripts/tests/run_python_test.py
@@ -110,6 +110,10 @@
)
]
+ if not runs:
+ raise Exception(
+ "No valid runs were found. Make sure you add runs to your file, see https://github.com/project-chip/connectedhomeip/blob/master/docs/testing/python.md document for reference/example.")
+
for run in runs:
print(f"Executing {run.py_script_path.split('/')[-1]} {run.run}")
main_impl(run.app, run.factoryreset, run.factoryreset_app_only, run.app_args,
diff --git a/src/python_testing/hello_test.py b/src/python_testing/hello_test.py
index d6953bf..01657bf 100644
--- a/src/python_testing/hello_test.py
+++ b/src/python_testing/hello_test.py
@@ -15,6 +15,13 @@
# limitations under the License.
#
+# test-runner-runs: run1
+# test-runner-run/run1/app: ${TYPE_OF_APP}
+# test-runner-run/run1/factoryreset: True
+# test-runner-run/run1/quiet: True
+# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
+# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
+
import logging
import chip.clusters as Clusters
