Skip to content

fix: retry failed model tests and summarize final results - #258

Open
chen2021673 wants to merge 2 commits into
masterfrom
fix/model-test-retries
Open

chen2021673 wants to merge 2 commits into
masterfrom
fix/model-test-retries

Conversation

@chen2021673

@chen2021673 chen2021673 commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

测例失败时自动重试一次,重试覆盖原日志;再次失败后记录结果并继续执行后续测例。

  • 运行结束时输出失败数 / 总测例数,高亮列出失败测例及日志路径。
  • CTest 按单个测例重试和统计,重试不重复计数,汇总无需临时文件。
  • 启动时完整解析脚本,避免运行期间修改文件导致 Bash 读取错位。

运行结果示例如下:
image

image

Comment thread scripts/run_models_and_profile.bash Outdated
echo "python3 write_to_feishu_sheet.py token.json --log-dir \"$(realpath "$RUN_OUTPUT_DIR")\""

print_test_summary
exit

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

print_test_summary 最后一条命令是 printf,都会返回 0,后面的裸 exit 会继承这个状态。即使有测试失败,脚本也返回 0;而 master 版本在同样场景返回 1。

建议

if (( ${#FAILED_TEST_CASES[@]} > 0 )); then
    exit 1
fi

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

修改:失败用例数量 >0 则显式 exit 1

local expanded_cmd="${cmd//\$LORA_WEIGHTS_DIR/$LORA_WEIGHTS_DIR}"
echo "[COMMAND] $expanded_cmd" >> "$log_path"
local attempt exit_code=0
for ((attempt=1; attempt<=max_attempts; attempt++)); do

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对于checkpoint 的用例可能存在一个风险场景,save → resume 成对用例:前面 save 用例失败后重试通过,如果有意外残留被 resume 用例当作恢复点读取,本应暴露的 checkpoint 写入/恢复问题可能被掩盖。
建议:attempt > 1 时清理该用例的 save 目标目录

@chen2021673 chen2021673 Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在 attempt > 1 时清理 checkpoint_dir,从当前用例命令中提取 --save=<dir>,清理后重跑 save 用例,避免第一次失败留下的部分 checkpoint 被后续 resume 用例误读。


print_test_summary
exit
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结尾这个的 } 配对的是文件第 3 行的 {,隔得有点远。建议在 } 上方加一行注释说明它是开头的闭合,增加阅读性

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在末尾 } 上方添加注释,说明它对应文件开头用于预加载脚本的 {。

Comment thread scripts/run_models_and_profile.bash Outdated
ctest --output-on-failure -LE cuda -j"$(nproc)"
mapfile -t cpu_tests < <(ctest -N -LE cuda | sed -n 's/^ *Test *#[0-9][0-9]*: //p')
local failed=0
if ! ctest --output-on-failure --repeat until-pass:2 -LE cuda -j"$(nproc)"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

首跑通过和“一挂一过”最终都记为 0,summary 看不出哪些用例不稳定。建议结果分三档 pass / pass-on-retry / fail,summary 加一行 Passed on retry: N。另外可以不写死 --repeat until-pass:2,可参数化 MAX_TEST_ATTEMPTS,默认 2。
另外端到端也可以记录一下哪些是重试 pass-on-retry 通过的。

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

修改:CTest 和端到端模型测试都会区分 pass、pass-on-retry、fail;重试次数通过 MAX_TEST_ATTEMPTS 配置,默认 2。
Summary 增加了总览和重试通过用例列表:

[SUMMARY] Pass / Pass on retry / Fail: X / Y / Z
[SUMMARY] Passed on retry: Y
[PASSED ON RETRY] ...

@chen2021673
chen2021673 force-pushed the fix/model-test-retries branch from 386c26b to 46b97f0 Compare November 20, 2026 07:31
- Add configurable MAX_TEST_ATTEMPTS for CTest and model test retries
- Classify results as pass, pass-on-retry, and fail in the summary
- List retry-passed cases and preserve nonzero exit status on failures
- Clean checkpoint save directories before retrying save cases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants