Skip to content

Commit eae343f

Browse files
Apollon77claude
Apollon77
and
claude
authored
feat(chip-testing): add AllDevicesTestApp mimic for spec-compliant per-device-type tests (#3687)
* feat(chip-testing): add getParameters helper for repeatable CLI flags Adds getParameters(name): string[] alongside existing getParameter/hasParameter/getIntParameter in GenericTestApp.ts. Returns all values for repeated -name <v> / --name <v> occurrences in argv order. Required by the upcoming AllDevicesTestApp's repeatable --device flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add DeviceTypeRegistry contract for AllDevicesApp Introduces EndpointHandle / DeviceTypeEntry interfaces and a Map-based registry with register/get/list helpers. Forms the seam between AllDevicesTestInstance (orchestrator) and per-device-type endpoint modules (each self-registers via side-effect import). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add RootEndpoint module with WiFi/Ethernet variants Lifts AllClustersTestInstance.setupServer()'s ServerNode.create body verbatim into buildRootNode(opts), exposing two seams: 1. NetworkCommissioning variant + networkCommissioning.networks switch on opts.wifi (Ethernet -> WiFi stub with empty networks list). 2. enableKeyHex sourced from opts instead of process.argv parse. All other root behaviors, basicInformation, productDescription, MDNS broadcast schedule, generalDiagnostics, localizationConfiguration, networkCommissioning details, operationalCredentials, timeFormatLocalization (incl. Buddhist calendar workaround), and userLabel are preserved exactly so AllDevicesTestApp inherits the same chip-test compatibility AllClustersTestApp has today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add on-off-light device type for AllDevicesApp First per-device-type endpoint module. Self-registers in DeviceTypeRegistry under the chip-CLI-compatible name "on-off-light" and constructs a single OnOffLightDevice endpoint. No backchannel — the spec doesn't require any test-injected commands for plain on/off lights. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add dimmable-light device type for AllDevicesApp Same pattern as on-off-light: plain DimmableLightDevice on the requested endpoint, no backchannel. Spec compliance comes from the device-type definition itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add contact-sensor device type with setBooleanState backchannel Initial booleanState.stateValue = false. Backchannel handles the same setBooleanState command that AllClustersTestInstance.backchannel() processes today (field name command.newState matches AllClusters convention). Endpoint-id filter prevents accidental cross-talk when multiple boolean-state-style devices share an AllDevicesTestApp instance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add water-leak-detector device type with setBooleanState backchannel Mirrors ContactSensorEndpoint shape but uses WaterLeakDetectorDevice so the device type identity matches the chip CLI string "water-leak-detector". Same setBooleanState backchannel — chip's harness sends the same command shape regardless of which boolean- state-style sensor is targeted; the endpoint-id filter routes correctly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add occupancy-sensor device type for AllDevicesApp OccupancySensorDevice requires explicit OccupancySensingServer feature selection per Matter 1.5 cluster definition; we pick PassiveInfrared (matches AllClustersTestInstance EP1 wiring). Initial occupied=false to mirror chip's TogglingOccupancySensorDevice baseline. No backchannel — AllClusters has no precedent handler for occupancy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add temperature-sensor device type for AllDevicesApp Initial measuredValue 2500 (25.00 degC, centi-degrees). No backchannel — the BackchannelCommand union does not currently include a setTemperature variant and AllClustersTestInstance has no precedent handler. A handler can be added later once the union is extended in @matter/testing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add device-type registration barrel for AllDevicesApp Side-effect imports trigger each per-device module's top-level registerDeviceType call so the registry is populated by the time AllDevicesTestInstance.setupServer runs. Alphabetical order. Adding a future device type (chime, soil-sensor, speaker once Matter 1.5.1 lands) is a one-line edit here plus a new module file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add AllDevicesTestInstance orchestrator Extends NodeTestInstance with: - parseDeviceArgs() — two-pass parser that respects CLI order while honoring explicit :N endpoint reservations. Spec example `--device contact-sensor --device on-off-light:5 --device occupancy-sensor` produces [contact-sensor:EP1, on-off-light:EP5, occupancy-sensor:EP2]. - setupServer() — builds root via buildRootNode (with --wifi seam and --enable-key passthrough), then resolves each spec to a registered factory and adds the constructed endpoint. Throws ValidationError with sorted listDeviceTypes() in the message for missing/unknown --device flags. - backchannel() — targeted-then-broadcast dispatch. command.endpointId addresses one endpoint first; remaining endpoints get a broadcast pass; unhandled commands fall through to NodeTestInstance.backchannel. Identity "alldevices-6100" keeps KVS namespace separate from AllClusters' "binford-6100". Side-effect import of devices/all-devices.js triggers all registerDeviceType calls before setupServer runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add AllDevicesTestApp bin entrypoint Mirrors AllClustersTestApp.ts shape: imports @matter/nodejs platform, sets process.title for stress-test detection, logs pid/argv for CI parsing, then delegates to startDeviceTestApp(AllDevicesTestInstance, StorageBackendAsyncJsonFile). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add AllDevicesTestApp.sh wrapper SIGTERM-handling wrapper sibling to AllClustersTestApp.sh. Tees stdout/stderr to test_alldevices.log, normalizes exit code 134 (abort) to 0 on SIGTERM, forwards all args to dist/esm/AllDevicesTestApp.js. Used by chip-tool-tests.yml jobs that expect a script entry rather than a node invocation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add chime device type for AllDevicesApp (Matter 1.5.1) ChimeServer requires installedChimeSounds (non-empty), selectedChime matching one of them, and enabled. Defaults to a single "Default" sound (chimeId=0) with playback enabled — sufficient for chip's chime test surface; tests that exercise sound selection can write a different selectedChime. Closes follow-up filed during initial design (TODO under matter-1.5/all-devices-app-chime.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add soil-sensor device type for AllDevicesApp (Matter 1.5.1) SoilMeasurement cluster requires both soilMoistureMeasurementLimits (a MeasurementAccuracy struct with at least one accuracyRange entry) and soilMoistureMeasuredValue. Defaults to a single 0-99% range with 5% percentMax accuracy, initial measuredValue 50% — chip's all-devices-app uses an IncreasingMoistureSoilSensorDevice that mutates this over time; our mimic provides a static baseline. Closes follow-up filed during initial design (TODO under matter-1.5/all-devices-app-soil-sensor.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): add speaker device type for AllDevicesApp (Matter 1.5.1) SpeakerDevice composes OnOffServer (mute) and LevelControlServer (volume) per the Matter 1.5.1 spec. No initial state needed — cluster defaults are valid. No backchannel — chip's LoggingSpeakerDevice in all-devices-app just logs; test coverage exercises the cluster commands directly. Closes follow-up filed during initial design (TODO under matter-1.5/all-devices-app-speaker.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): register chime/soil-sensor/speaker in AllDevicesApp barrel Adds the 3 newly implemented device types to the side-effect-import barrel so they self-register at AllDevicesTestInstance startup. Matches chip's all-devices-app supported set 1:1 (9 device types total). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): satisfy WiFi NetworkCommissioning init constraints in AllDevices RootEndpoint The 1.5.1 NetworkCommissioning cluster requires non-empty supportedWiFiBands (min 1) plus scanMaxTimeSeconds and connectMaxTimeSeconds when WiFi feature is selected. Without these, the WiFi-variant root fails behavior validation at ServerNode.create with "Array length 0 is not within bounds" on supportedWiFiBands. Add minimal stub values (2.4GHz band, 1s timeouts) so --wifi smoke passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(chip-testing): route all-devices target to new AllDevicesTestApp Switches all 10 --app-path all-devices: mappings in chip-tool-tests.yml from AllClustersTestApp.{js,sh} to AllDevicesTestApp.{js,sh}. The matter.js side of the all-devices test target now exercises the spec-compliant per-device-type mimic instead of the kitchen-sink AllClusters EP1 it shared with all-clusters. all-clusters: mappings are untouched. chip-matterjs-tests.yml:113 (chip-native bin/all-devices-app) is also untouched — that workflow tests matter.js controller against chip's own device binary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): mirror AllClusters TCP fields in AllDevices RootEndpoint AllClustersTestInstance's network block sets `tcp: true` and a TEST_PREFER_TCP-driven `transportPreference`. Without these, the new AllDevicesTestApp silently runs UDP-only even when the matterjs-tests-core-tcp CI job exports TEST_PREFER_TCP=1 — that job's `--app-path all-devices:` now points at AllDevicesTestApp, so the bug would let TCP transport tests pass-by-skip rather than actually exercising TCP. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(chip-testing): cat AllDevices log on TestSelfFabricRemoval failure The two TestSelfFabricRemoval invocations in chip-tool-tests.yml fall back to cat ./test_allclusters.log on failure. Now that AllDevicesTestApp.sh writes to ./test_alldevices.log and the all-devices test target maps to that script, also cat ./test_alldevices.log so a failure originating from AllDevices is debuggable in CI logs (otherwise we'd see the AllClusters log alone, likely unrelated to the actual failure). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): throw on missing value in getParameters Per Copilot PR review: silently dropping a trailing -name/--name token (no value following) makes "--device required" errors confusing when the user actually typed --device. Now throws ValidationError with the parameter name, matching chip CLI parity expectations. Existing single-value getParameter retains its lenient behavior to avoid churn in callers that rely on it; the strict throw applies only to the new repeatable helper used by --device. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): enforce EndpointNumber upper bound in --device parser Per Copilot PR review: parseDeviceArgs validated ep >= 1 but not ep <= 0xFFFE, so --device foo:70000 would surface a generic out-of-bounds throw from the EndpointNumber() brand at later use rather than the token-specific "Invalid endpoint in --device" message. Add the upper-bound check next to the existing lower-bound check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): correct exit code capture in AllDevicesTestApp.sh SIGTERM handler Per Copilot PR review: when SIGTERM arrives before CHILD_PID is assigned, the prior implementation captured \$? from the [ -n "\$CHILD_PID" ] test (status 1) rather than from a wait, so the wrapper would exit 1 even though no child failed. Initialize EXIT_CODE=0 up front and only overwrite from wait when a child was actually waited for. (The same bug exists in AllClustersTestApp.sh — left for a separate follow-up to keep this change scoped.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): correct exit code capture in AllClustersTestApp.sh SIGTERM handler Same bug we just fixed in AllDevicesTestApp.sh: when SIGTERM arrives before CHILD_PID is assigned, the prior code captured \$? from the [ -n "\$CHILD_PID" ] test rather than wait, so the wrapper exited 1 with no child failure. Initialize EXIT_CODE=0 up front and only overwrite from wait when a child was actually waited for. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(testing): extend Subject.Factory with optional appArgs Adds a second optional parameter Subject.Options to the Factory signature so per-run app-args (chip header `app-args:`) can flow into in-process subjects without going through process.argv. Existing factories that ignore the extra arg remain compatible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(testing): add chip.subjectFor app-string→factory registry Introduces a Map<string, Subject.Factory> keyed by chip-test-header app names ("all-clusters", "all-devices", "bridge", "tv", "rvc", future). Setup-time registration via chip.subjectFor("name", factory). Duplicate registration throws. Task 3 will wire the registry into defineTest dispatch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(testing): dispatch chip tests by descriptor.app and forward app-args Wires the runtime side of the chip-test app registry: - parseAppArgs() splits descriptor.config["app-args"] (string or array form). - defineTest's beforeOne resolves the factory by priority: explicit .subject() override on the builder, then State.subjectForApp(descriptor.app), then chip.defaultSubject. A descriptor that names an unregistered app fails loud with a message naming the missing app and pointing at chip.subjectFor. - activateSubject + loadSubject thread appArgs through. The subject cache key becomes (factory, kind, appArgs) so distinct per-run app-args produce distinct cached subjects (chip multi-run tests with different --device flags). Backward compatible: tests without descriptor.app keep using chip.defaultSubject. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): plumb appArgs through DeviceTestInstanceConfig Adds an optional appArgs?: string[] field to DeviceTestInstanceConfig so the support.ts factory can forward chip-framework Subject.Options.appArgs into the TestInstance constructor. Subjects that need runtime CLI-style configuration (AllDevicesTestApp's --device list) consume it via this.config.appArgs in setupServer(). Existing TestInstance classes (AllClusters/Bridge/Tv/Rvc) ignore the field — change is purely additive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(chip-testing): AllDevicesTestInstance reads device list from config first Replaces parseDeviceArgs() (which read process.argv directly via getParameters/ hasParameter/getParameter) with parseRuntimeArgs(args[]) — same logic but argv-source-independent. Constructor extracts config.appArgs into a private field; setupServer() uses this.#appArgs ?? process.argv.slice(2) as the source. Behavior unchanged for standalone CLI runs (chip-tool-tests CI binary, local smoke). Test framework now flows per-run --device/--wifi/--enable-key into the subject via Subject.Options.appArgs, unblocking chip multi-run python tests where each run targets a different device type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(chip-testing): register all subjects via chip.subjectFor for descriptor-app dispatch Adds AllDevicesApp and registers all 5 subjects (all-clusters, all-devices, bridge, tv, rvc) under their normalized chip-header names. The App() helper now also threads Subject.Options.appArgs into the constructor so per-run chip-header app-args (e.g. "--device on-off-light:1") flow into the in-process subject. chip.defaultSubject stays at AllClustersApp for tests without descriptor.app. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(testing): scope app-args to descriptor-app dispatch only CI regression: forwarding parseAppArgs() output for every test (including tests that fall through to chip.defaultSubject) caused per-test cache key divergence — chip headers carry per-test --trace-to / --KVS values, so loadSubject() spawned a fresh subject for every test, thrashing MDNS and producing CASE timeouts on subsequent tests (TC_WHM_2_1, TC_DRLK_*, etc). Only set appArgs when descriptor.app is at top-level (= multi-run suite member explicitly dispatched via chip.subjectFor). Tests using chip.defaultSubject keep their pre-PR cache behavior (kind only). Also enhances print-report to surface descriptor.app per test in brackets, which made the bug easy to spot during inspect. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chip-testing): pass full method name to get_test_desc in descriptor generator chip's MatterBaseTest.get_test_desc(test) does getattr(self, test) and requires the full method name (test_TC_FOO_2_3). The legacy short-form aliases ("FOO_2_3") that made our previous call signature work have been removed upstream — TC_CGEN_2_4 now raises AttributeError, which abort-aborts the descriptor build and leaves the chip docker image with a stale /lib/test-descriptor.json (no recent multi-run / new app references, no chime/soil/speaker, etc). Pass test_method_name instead of the stripped form. Wrap in try/except so a single bad test class can't break the whole image build going forward. [rebuild-chip] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(testing): skip multi-run tests targeting unregistered apps instead of erroring When a multi-run python test descriptor names an app we have no Subject.Factory for (e.g. \${LOCK_APP}, \${LIT_ICD_APP}, \${WATER_HEATER_APP}), skip just that run rather than throwing. Chip CI lists more apps than we mimic; whitelist semantics let the runs we DO support proceed without surfacing failures from runs we deliberately don't cover. Other runs of the same test continue normally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(testing): drop appArgs from subject cache key to keep fabric stable Including appArgs in loadSubject's cache key spawned a fresh in-process subject for each multi-run member of a chip python test that shared a factory but differed in script-side flags (e.g. TC_SWTCH run1/run2/run3 all use ALL_CLUSTERS_APP but pass different --app-pipe paths). Each new subject was commissioned independently — chip controller storage held one fabric while the restored subject snapshot held another, producing the symptom CHIP: Compressed FabricId 0x6BB913D815052317 matter.js: mdns:074A2057D2EE4E7C-... and CASE Avahi resolve timeouts. Cache by (factory, kind) only. appArgs are still forwarded on first construct (needed for AllDevicesTestApp's --device list); cache hits reuse the cached subject's commissioning state so all multi-run members of a single factory share fabric and chip controller storage stays consistent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(chip-testing): bump chip pin + show config.app in test output - support/chip/sha.txt: 6891456 (Apr 21) -> e8152e9 (today). The Apr 21 pin predated chip's TC_OO_2_7 multi-run addition (Apr 23) plus all the recent chime/soil/speaker/etc test additions, so our descriptor never picked them up. - print-report.ts: surface descriptor.config.app in the [...] tag when descriptor.app (top-level, set only for multi-run suite members) is absent. Single-run python tests now also display the app they target. [rebuild-chip] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(chip-testing): pin chip just before today's TC_LVL_9_1 step0c addition e8152e9 (today's master) added a step0c label to TC_LVL_9_1, which the matter.js LevelControl behavior doesn't yet satisfy and surfaced as the only test-app-slow failure on PR #3687. Roll the pin back one commit to ee82434 (also today, just the prior commit) — keeps the multi-run TC_OO_2_7 (added Apr 23) and chime/soil/ speaker definitions, drops the new step0c assertion. TC_LVL_9_1 step0c is follow-up matter.js work. [rebuild-chip] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(chip-testing): add diag job running TC_LVL_9_1 with --all-logs Adds a diagnostic-only test file (test/diag/LVL_9_1.test.ts), an npm test-diag script, and a continue-on-error CI job in build-test.chip.yml. The job runs only LVL/9.1 with --all-logs to capture the full chip subprocess output so we can see why TC_LVL_9_1 fails against newer chip pins (annotation- level CI logs swallow chip stderr/stdout). [rebuild-chip] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(chip-testing): bump chip pin to 501ae86 with TC_LVL_9_1 main entry fix Upstream chip landed the missing \`if __name__ == "__main__": default_matter_test_main()\` on TC_LVL_9_1.py, so the test now actually runs when invoked directly. Bump the pin and trigger an image rebuild to verify. [rebuild-chip] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(chip-testing): drop diag scaffolding now that TC_LVL_9_1 runs Upstream chip added the missing default_matter_test_main() entry; the test-diag job confirmed TC_LVL_9_1 runs against the new chip pin. Remove the diagnostic test file, npm script, and CI job — served their purpose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 808b70b commit eae343f

25 files changed

Lines changed: 877 additions & 31 deletions

‎.github/workflows/chip-tool-tests.yml‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -180,12 +180,12 @@ jobs:
180180
--iterations 1 \
181181
--tool-path chip-tool:../bin/chip-tool \
182182
--app-path all-clusters:../support/chip-testing/AllClustersTestApp.sh \
183-
--app-path all-devices:../support/chip-testing/AllClustersTestApp.sh \
183+
--app-path all-devices:../support/chip-testing/AllDevicesTestApp.sh \
184184
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
185185
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
186186
--app-path rvc:../support/chip-testing/AllClustersTestApp.sh \
187187
--app-path lit-icd:../support/chip-testing/AllClustersTestApp.sh \
188-
|| (cat ./test_allclusters.log && exit 1)'
188+
|| (cat ./test_allclusters.log; cat ./test_alldevices.log; exit 1)'
189189
./scripts/run_in_build_env.sh \
190190
'./scripts/tests/run_test_suite.py \
191191
--runner chip_tool_python \
@@ -195,7 +195,7 @@ jobs:
195195
--iterations 1 \
196196
--tool-path chip-tool:../bin/chip-tool \
197197
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
198-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
198+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
199199
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
200200
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
201201
--app-path rvc:../support/chip-testing/AllClustersTestApp.sh \
@@ -231,12 +231,12 @@ jobs:
231231
--iterations 1 \
232232
--tool-path chip-tool:../support/chip-testing/dist/esm/ControllerWebSocketTestApp.js \
233233
--app-path all-clusters:../support/chip-testing/AllClustersTestApp.sh \
234-
--app-path all-devices:../support/chip-testing/AllClustersTestApp.sh \
234+
--app-path all-devices:../support/chip-testing/AllDevicesTestApp.sh \
235235
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
236236
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
237237
--app-path rvc:../support/chip-testing/AllClustersTestApp.sh \
238238
--app-path lit-icd:../support/chip-testing/AllClustersTestApp.sh \
239-
|| (cat ./test_allclusters.log && exit 1)'
239+
|| (cat ./test_allclusters.log; cat ./test_alldevices.log; exit 1)'
240240
./scripts/run_in_build_env.sh \
241241
'./scripts/tests/run_test_suite.py \
242242
--runner chip_tool_python \
@@ -247,7 +247,7 @@ jobs:
247247
--iterations 1 \
248248
--tool-path chip-tool:../support/chip-testing/dist/esm/ControllerWebSocketTestApp.js \
249249
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
250-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
250+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
251251
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
252252
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
253253
--app-path rvc:../support/chip-testing/AllClustersTestApp.sh \
@@ -285,7 +285,7 @@ jobs:
285285
--iterations 1 \
286286
--tool-path chip-tool:../support/chip-testing/dist/esm/ControllerWebSocketTestApp.js \
287287
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
288-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
288+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
289289
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
290290
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
291291
--app-path rvc:../support/chip-testing/AllClustersTestApp.sh \
@@ -322,7 +322,7 @@ jobs:
322322
--iterations 1 \
323323
--tool-path chip-tool:../bin/chip-tool \
324324
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
325-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
325+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
326326
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
327327
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
328328
'
@@ -356,7 +356,7 @@ jobs:
356356
--iterations 1 \
357357
--tool-path chip-tool:../support/chip-testing/dist/esm/ControllerWebSocketTestApp.js \
358358
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
359-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
359+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
360360
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
361361
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
362362
--app-path energy-gateway:../support/chip-testing/dist/esm/AllClustersTestApp.js \
@@ -394,7 +394,7 @@ jobs:
394394
--iterations 1 \
395395
--tool-path chip-tool:../bin/chip-tool \
396396
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
397-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
397+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
398398
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
399399
--app-path tv:../support/chip-testing/dist/esm/TvTestApp.js \
400400
'
@@ -427,7 +427,7 @@ jobs:
427427
run \
428428
--tool-path chip-tool:../bin/chip-tool \
429429
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
430-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
430+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
431431
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
432432
'
433433
matterjs-tests-long:
@@ -458,7 +458,7 @@ jobs:
458458
run \
459459
--tool-path chip-tool:../support/chip-testing/dist/esm/ControllerWebSocketTestApp.js \
460460
--app-path all-clusters:../support/chip-testing/dist/esm/AllClustersTestApp.js \
461-
--app-path all-devices:../support/chip-testing/dist/esm/AllClustersTestApp.js \
461+
--app-path all-devices:../support/chip-testing/dist/esm/AllDevicesTestApp.js \
462462
--app-path bridge:../support/chip-testing/dist/esm/BridgeTestApp.js \
463463
'
464464

‎packages/testing/src/chip/chip.ts‎

Lines changed: 55 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,14 @@ export interface Chip {
6565
*/
6666
defaultSubject: Subject.Factory;
6767

68+
/**
69+
* Register a {@link Subject.Factory} for a chip-test-header app name (e.g. `"all-clusters"`,
70+
* `"all-devices"`). When a test descriptor has an `app` field set by the descriptor generator,
71+
* the framework dispatches to the registered factory automatically. Manual `.subject(...)`
72+
* overrides on a builder still take precedence over the registry.
73+
*/
74+
subjectFor(app: string, factory: Subject.Factory): void;
75+
6876
/**
6977
* Clear the MDNS cache.
7078
*/
@@ -292,11 +300,36 @@ function createBuilder(initial: {
292300

293301
// We do this separately from the test itself because we don't want activation to appear as part of the test if
294302
// it fails
295-
beforeOne(mochaTest, async () =>
296-
State.activateSubject(subject ?? State.subject, startCommissioned, test, (subject, test) =>
297-
runBeforeHooks(beforeStartHooks, subject, test),
298-
),
299-
);
303+
beforeOne(mochaTest, async function (this: Mocha.Context) {
304+
// Resolution order: explicit .subject() override on the builder, then descriptor.app
305+
// (chip-test-header app for multi-run suite members), then chip.defaultSubject. A
306+
// multi-run member that names an unregistered app is skipped (other runs of the same
307+
// test continue) — chip CI lists more apps than we mimic, and ignoring those runs is
308+
// the whitelist behavior we want.
309+
let factory = subject;
310+
let appArgs: string[] | undefined;
311+
if (factory === undefined && descriptor.app !== undefined) {
312+
factory = State.subjectForApp(descriptor.app);
313+
if (factory === undefined) {
314+
this.skip();
315+
}
316+
// app-args are scoped to the named app; forwarding them to defaultSubject would
317+
// pollute the subject cache with per-test variants (e.g. --trace-to paths) and
318+
// thrash setup on tests that don't need per-run dispatch.
319+
appArgs = parseAppArgs(descriptor);
320+
}
321+
if (factory === undefined) {
322+
factory = State.subject;
323+
}
324+
325+
await State.activateSubject(
326+
factory,
327+
startCommissioned,
328+
test,
329+
(subject, test) => runBeforeHooks(beforeStartHooks, subject, test),
330+
appArgs,
331+
);
332+
});
300333

301334
mochaTest.descriptor = test.descriptor;
302335
implementations.set(descriptor, mochaTest);
@@ -305,6 +338,17 @@ function createBuilder(initial: {
305338
}
306339
}
307340

341+
function parseAppArgs(descriptor: TestDescriptor): string[] | undefined {
342+
const raw = descriptor.config?.["app-args"];
343+
if (raw === undefined) return undefined;
344+
if (Array.isArray(raw)) return raw.map(String);
345+
if (typeof raw === "string") {
346+
const trimmed = raw.trim();
347+
return trimmed === "" ? undefined : trimmed.split(/\s+/);
348+
}
349+
return undefined;
350+
}
351+
308352
function chipFn(subjectOrFirstInclusion: Subject.Factory | string | undefined, ...include: string[]): chip.Builder {
309353
if (typeof subjectOrFirstInclusion === "function") {
310354
return chip()
@@ -326,6 +370,12 @@ Object.defineProperties(chipFn, {
326370
},
327371
},
328372

373+
subjectFor: {
374+
value: (app: string, factory: Subject.Factory) => {
375+
State.registerSubjectForApp(app, factory);
376+
},
377+
},
378+
329379
activeTest: {
330380
get: () => State.test,
331381
},

‎packages/testing/src/chip/state.ts‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ const Values = {
4545
singleUseSubject: false,
4646
closers: Array<() => Promise<void>>(),
4747
subjects: new Map<Subject.Factory, Record<string, Subject>>(),
48+
subjectsByApp: new Map<string, Subject.Factory>(),
4849
snapshots: new Map<Subject, {}>(),
4950
containerLifecycleInstalled: false,
5051
testMap: new Map<TestDescriptor, Test>(),
@@ -82,6 +83,17 @@ export const State = {
8283
return subject;
8384
},
8485

86+
registerSubjectForApp(app: string, factory: Subject.Factory) {
87+
if (Values.subjectsByApp.has(app)) {
88+
throw new Error(`Subject factory already registered for app "${app}"`);
89+
}
90+
Values.subjectsByApp.set(app, factory);
91+
},
92+
93+
subjectForApp(app: string): Subject.Factory | undefined {
94+
return Values.subjectsByApp.get(app);
95+
},
96+
8597
get pullBeforeTesting() {
8698
return Values.pullBeforeTesting;
8799
},
@@ -277,14 +289,15 @@ export const State = {
277289
startCommissioned: boolean,
278290
test: Test,
279291
beforeStart?: chip.BeforeHook,
292+
appArgs?: string[],
280293
) {
281294
let subject;
282295
if (startCommissioned) {
283296
// We cache commissioned subjects
284-
subject = loadSubject(factory, test.descriptor.kind);
297+
subject = loadSubject(factory, test.descriptor.kind, appArgs);
285298
} else {
286299
// No need to cache uncommissioned subjects
287-
subject = factory(test.descriptor.kind ?? "unknown");
300+
subject = factory(test.descriptor.kind ?? "unknown", { appArgs });
288301
}
289302

290303
if (Values.activeSubject === subject) {
@@ -572,16 +585,20 @@ async function configureLocalController() {
572585

573586
/**
574587
* Obtain a subject. Subjects are qualified by factory and test domain.
588+
*
589+
* appArgs are forwarded to the factory only on first construct — subsequent restores reuse the cached subject so
590+
* commissioning/fabric stays stable across runs that share a factory (chip multi-run tests often differ only in
591+
* script-side flags like --app-pipe / --test-case which the in-process subject does not care about).
575592
*/
576-
function loadSubject(factory: Subject.Factory, kind: TestDescriptor["kind"]) {
593+
function loadSubject(factory: Subject.Factory, kind: TestDescriptor["kind"], appArgs?: string[]) {
577594
let forFactory = Values.subjects.get(factory);
578595
if (forFactory === undefined) {
579596
Values.subjects.set(factory, (forFactory = {}));
580597
}
581598

582599
let subject = forFactory[kind];
583600
if (subject === undefined) {
584-
subject = forFactory[kind] = factory(kind);
601+
subject = forFactory[kind] = factory(kind, { appArgs });
585602
}
586603

587604
return subject;

‎packages/testing/src/device/subject.ts‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,14 +28,22 @@ export namespace Subject {
2828
/**
2929
* Producer for test subjects.
3030
*
31-
* We cache subjects based on the subject factory although a factory may be invoked multiple times if the subject
32-
* initializes differently for different test implementations.
31+
* Subjects are cached per (factory, domain, appArgs). A factory may be invoked multiple times if the subject
32+
* initializes differently for different test implementations (e.g. chip multi-run tests with distinct app-args).
3333
*/
3434
export interface Factory {
35-
(domain: string): Subject;
35+
(domain: string, options?: Subject.Options): Subject;
3636
pics?: PicsFile;
3737
}
3838

39+
/**
40+
* Per-invocation overrides for a {@link Subject.Factory}. Used to forward chip header `app-args:` into the
41+
* in-process subject without going through process.argv.
42+
*/
43+
export interface Options {
44+
appArgs?: string[];
45+
}
46+
3947
export type CommissioningMethod = "onnetwork";
4048

4149
export interface WifiNetwork {

‎packages/testing/src/print-report.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,11 @@ function printDescriptor(descriptor: TestDescriptor, includeDisabled?: boolean,
8080
title = `${title} ${colors.cyan(`(${steps} steps)`)}`;
8181
}
8282

83+
const app = member.app ?? (member.config?.["app"] as string | undefined);
84+
if (app) {
85+
title = `${title} ${colors.magenta(`[${app}]`)}`;
86+
}
87+
8388
if (member.pics) {
8489
try {
8590
const expr = new PicsExpression(member.pics);

‎support/chip-testing/AllClustersTestApp.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,12 @@ CHILD_PID=""
1616
# Function to handle SIGTERM
1717
sigterm_handler() {
1818
echo "$(date '+%Y-%m-%d %H:%M:%S') - Received SIGTERM, forwarding to child process (PID: $CHILD_PID)" | tee -a "$LOG_FILE"
19+
EXIT_CODE=0
1920
if [ -n "$CHILD_PID" ]; then
2021
kill -TERM "$CHILD_PID" 2>/dev/null
2122
wait "$CHILD_PID"
23+
EXIT_CODE=$?
2224
fi
23-
EXIT_CODE=$?
2425

2526
# If the child reported 134 (abort), normalize to 0 for the SIGTERM case
2627
if [ "$EXIT_CODE" -eq 134 ]; then

‎support/chip-testing/AllDevicesTestApp.sh‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
3+
# Wrapper script for alldevicesapp
4+
# Captures stdout/stderr to log file and handles SIGTERM
5+
6+
# Path to the real executable
7+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
8+
REAL_PROGRAM="$SCRIPT_DIR/dist/esm/AllDevicesTestApp.js"
9+
10+
# Log file (hardcoded)
11+
LOG_FILE="./test_alldevices.log"
12+
13+
# Variable to store the PID of the wrapped process
14+
CHILD_PID=""
15+
16+
# Function to handle SIGTERM
17+
sigterm_handler() {
18+
echo "$(date '+%Y-%m-%d %H:%M:%S') - Received SIGTERM, forwarding to child process (PID: $CHILD_PID)" | tee -a "$LOG_FILE"
19+
EXIT_CODE=0
20+
if [ -n "$CHILD_PID" ]; then
21+
kill -TERM "$CHILD_PID" 2>/dev/null
22+
wait "$CHILD_PID"
23+
EXIT_CODE=$?
24+
fi
25+
26+
# If the child reported 134 (abort), normalize to 0 for the SIGTERM case
27+
if [ "$EXIT_CODE" -eq 134 ]; then
28+
echo "$(date '+%Y-%m-%d %H:%M:%S') - Normalizing exit code 134 -> 0 after SIGTERM" | tee -a "$LOG_FILE"
29+
EXIT_CODE=0
30+
fi
31+
32+
# Log completion
33+
echo "$(date '+%Y-%m-%d %H:%M:%S') - alldevicesapp exited with code after SIGTERM: $EXIT_CODE" | tee -a "$LOG_FILE"
34+
35+
# Exit with the same code as the wrapped program
36+
exit $EXIT_CODE
37+
}
38+
39+
# Set up SIGTERM trap
40+
trap sigterm_handler SIGTERM
41+
42+
# Log start
43+
echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting alldevicesapp with arguments: $@" | tee -a "$LOG_FILE"
44+
45+
# Run the real program with all arguments, capturing stdout and stderr
46+
# Using process substitution to tee both streams
47+
"$REAL_PROGRAM" "$@" > >(tee -a "$LOG_FILE") 2> >(tee -a "$LOG_FILE" >&2) &
48+
49+
# Store the PID
50+
CHILD_PID=$!
51+
52+
# Wait for the process to complete
53+
wait "$CHILD_PID"
54+
EXIT_CODE=$?
55+
56+
# Log completion
57+
echo "$(date '+%Y-%m-%d %H:%M:%S') - alldevicesapp exited with code: $EXIT_CODE" | tee -a "$LOG_FILE"
58+
59+
# Exit with the same code as the wrapped program
60+
exit $EXIT_CODE

‎support/chip-testing/src/AllDevicesTestApp.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/usr/bin/env node
2+
/**
3+
* @license
4+
* Copyright 2022-2026 Matter.js Authors
5+
* SPDX-License-Identifier: Apache-2.0
6+
*/
7+
8+
import "@matter/nodejs";
9+
10+
import { AllDevicesTestInstance } from "./AllDevicesTestInstance.js";
11+
import { startDeviceTestApp } from "./GenericTestApp.js";
12+
import { StorageBackendAsyncJsonFile } from "./storage/StorageBackendAsyncJsonFile.js";
13+
14+
process.title = "AllDevicesTestApp.js"; // Needed for Stress testing to detect the process to kill.
15+
16+
console.log("Start AllDevicesApp");
17+
console.log(process.pid);
18+
console.log(process.argv);
19+
20+
startDeviceTestApp(AllDevicesTestInstance, StorageBackendAsyncJsonFile).catch(console.error);

0 commit comments

Comments
 (0)