Skip to content

rescan-tests: report added, modified and removed test cases - #139

Merged
rquidute merged 9 commits into
project-chip:v2.17-cli-developfrom
rquidute:feature/1200-rescan-report-changes
Nov 7, 2026
Merged

rquidute merged 9 commits into
project-chip:v2.17-cli-developfrom
rquidute:feature/1200-rescan-report-changes

Conversation

@rquidute

@rquidute rquidute commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1200

Summary

th-cli rescan-tests renders the changes returned by the backend rescan (see project-chip/certification-tool-backend#413):

  • test cases added / modified / removed, grouped by collection, plus No changes in: ... for the rest
  • Rescanned test collections successfully (N found). No changes detected. when nothing changed
  • scripts that failed to parse, listed with the reason
  • falls back to the plain summary if the backend response has no changes

Notes

th_cli/api_lib_autogen was updated by hand to match the generator output for the new backend schema (changes on TestCollections); regenerate with scripts/generate_client.sh once the backend PR is merged.

Testing

tests/test_rescan_tests.py: 12 passed (changes grouped by collection, no changes, parse errors, old backend without changes).

Also prints changes.custom_files from the backend as a "Custom files" block: the files added, modified or removed in custom/ since the previous rescan, with (replaces sdk/X) on the ones that replace an SDK module. A change limited to custom files is not reported as "No changes detected". Nothing extra is printed when the list is empty or the backend does not send the field. Needs project-chip/certification-tool-backend#413.

The added, modified and removed lines are colored green, yellow and red (colorize_change in colorize.py). The +, ~, - markers and labels stay, TH_CLI_NO_COLOR is honored, and the codes are stripped when the output is not a terminal.

Render the `changes` returned by the backend rescan: test cases added,
modified and removed grouped by collection, the collections without changes,
and the scripts that failed to parse with the reason. A rescan without
changes now says so explicitly.

The autogen models and rescan return type are updated by hand to match the
backend response; regenerate them with scripts/generate_client.sh against
the updated backend.

Part of #1200
Match the backend, which now returns the change report on TestCollections
instead of a separate response model (that made datamodel-codegen emit an
invalid `class TestCollections(TestCollections2)`). Fall back to the plain
summary when the backend response has no `changes`.

Part of #1200
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Explain Change →

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 65894e89-a646-4838-9d28-4709f135578e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The response models now support optional rescan change reports, including added, modified, and removed test cases, affected collections, and script errors. The rescan command passes its response to a report function. The function prints the collection count and formats available changes or errors. It also handles responses without a change report. Tests cover these output cases.

Sequence Diagram(s)

sequenceDiagram
  participant RescanCommand
  participant RescanAPI
  participant print_rescan_report
  participant Terminal
  RescanCommand->>RescanAPI: Request rescan
  RescanAPI-->>RescanCommand: TestCollections response
  RescanCommand->>print_rescan_report: Pass response
  print_rescan_report->>Terminal: Print summary and report
Loading

Severity of issue fixed: Medium

Merge Risk: Minimal · previous review 82b3a

No actionable merge-blocking risk is established for the rescan reporting changes. Merge after normal checks pass.

Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Linked Issues check Inconclusive [#1200] The CLI groups added, modified, and removed test cases by collection, reports no changes and parse errors, and supports responses without changes. CLI tests cover these cases. The reviewed r… Provide reviewable backend implementation and test evidence, and the current User Guide section for #1200.
Passed Passed checks: 4 passed

Check name Status Explanation
Title check Passed The title accurately describes the main change: reporting added, modified, and removed test cases.
Out of Scope Changes check Passed [#1200] The CLI report, response models, and rendering tests support the rescan reporting objective. No unrelated changes are evident in the whole-PR summary.
Docstring Coverage Passed Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentati…
Description check Passed The description matches the rescan change reporting, model updates, and tests. Its custom-file and color claims are not supported by the supplied change summaries, but the description is related to th…
Full details: Docstring Coverage

Explanation

Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 3 files.

Full details: Description check

Explanation

The description matches the rescan change reporting, model updates, and tests. Its custom-file and color claims are not supported by the supplied change summaries, but the description is related to the changeset.

Full details: Linked Issues check

Explanation

[#1200] The CLI groups added, modified, and removed test cases by collection, reports no changes and parse errors, and supports responses without changes. CLI tests cover these cases. The reviewed repository does not contain the backend or the User Guide. The backend diff logic, its tests, preservation of test_collections, the definition of modified, and the User Guide update cannot be verified from the available evidence.

Resolution

Provide reviewable backend implementation and test evidence, and the current User Guide section for #1200.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b325742d-4c28-454b-a1fb-2e4319864d96
Commits

Reviewing files that changed from the base of the PR and between 945a054 and 82b3af8.

Files selected for processing: 3

  • tests/test_rescan_tests.py
  • th_cli/api_lib_autogen/models.py
  • th_cli/commands/rescan_tests.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@mergify

mergify Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@rquidute rquidute self-assigned this Nov 6, 2026
@rquidute
rquidute requested a review from antonio-amjr November 6, 2026 13:34
Comment thread th_cli/commands/rescan_tests.py Resolved
Print the backend's changes.custom_overrides: custom/*.py files that replace an
SDK module of the same name (used by every test that imports them), and, apart,
same-name copies that nothing imports. A backend without the field, or with
empty lists, prints nothing extra.

The models are edited by hand to match the backend schema; regenerate with
scripts/generate_client.sh once the backend change is merged.

Refs project-chip/certification-tool#1200
Print the backend's changes.custom_files as a "Custom files" block: files added,
modified or removed in custom/, with "(replaces sdk/X)" on the ones that replace
an SDK module. Replaces the two override lists, which listed same-name copies of
SDK tests that replace nothing. A change limited to custom files, a helper for
example, is no longer reported as "No changes detected".

The models are edited by hand to match the backend schema; regenerate with
scripts/generate_client.sh once the backend change is merged.

Refs project-chip/certification-tool#1200
Color the added, modified and removed lines green, yellow and red, for test cases
and custom files. The +, ~ and - markers and the labels stay, so the output reads
the same without color. Honors TH_CLI_NO_COLOR and is stripped when the output is
not a terminal.

Refs project-chip/certification-tool#1200
@rquidute
rquidute merged commit 483dac2 into project-chip:v2.17-cli-develop Nov 7, 2026
6 checks passed
@rquidute
rquidute deleted the feature/1200-rescan-report-changes branch November 7, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants