Repository navigation
fix: retry failed model tests and summarize final results - #258
chen2021673 wants to merge 2 commits into
Conversation
| echo "python3 write_to_feishu_sheet.py token.json --log-dir \"$(realpath "$RUN_OUTPUT_DIR")\"" | ||
|
|
||
| print_test_summary | ||
| exit |
There was a problem hiding this comment.
print_test_summary 最后一条命令是 printf,都会返回 0,后面的裸 exit 会继承这个状态。即使有测试失败,脚本也返回 0;而 master 版本在同样场景返回 1。
建议
if (( ${#FAILED_TEST_CASES[@]} > 0 )); then
exit 1
fi
There was a problem hiding this comment.
修改:失败用例数量 >0 则显式 exit 1
| local expanded_cmd="${cmd//\$LORA_WEIGHTS_DIR/$LORA_WEIGHTS_DIR}" | ||
| echo "[COMMAND] $expanded_cmd" >> "$log_path" | ||
| local attempt exit_code=0 | ||
| for ((attempt=1; attempt<=max_attempts; attempt++)); do |
There was a problem hiding this comment.
对于checkpoint 的用例可能存在一个风险场景,save → resume 成对用例:前面 save 用例失败后重试通过,如果有意外残留被 resume 用例当作恢复点读取,本应暴露的 checkpoint 写入/恢复问题可能被掩盖。
建议:attempt > 1 时清理该用例的 save 目标目录
There was a problem hiding this comment.
在 attempt > 1 时清理 checkpoint_dir,从当前用例命令中提取 --save=<dir>,清理后重跑 save 用例,避免第一次失败留下的部分 checkpoint 被后续 resume 用例误读。
|
|
||
| print_test_summary | ||
| exit | ||
| } |
There was a problem hiding this comment.
结尾这个的 } 配对的是文件第 3 行的 {,隔得有点远。建议在 } 上方加一行注释说明它是开头的闭合,增加阅读性
There was a problem hiding this comment.
在末尾 } 上方添加注释,说明它对应文件开头用于预加载脚本的 {。
| ctest --output-on-failure -LE cuda -j"$(nproc)" | ||
| mapfile -t cpu_tests < <(ctest -N -LE cuda | sed -n 's/^ *Test *#[0-9][0-9]*: //p') | ||
| local failed=0 | ||
| if ! ctest --output-on-failure --repeat until-pass:2 -LE cuda -j"$(nproc)"; then |
There was a problem hiding this comment.
首跑通过和“一挂一过”最终都记为 0,summary 看不出哪些用例不稳定。建议结果分三档 pass / pass-on-retry / fail,summary 加一行 Passed on retry: N。另外可以不写死 --repeat until-pass:2,可参数化 MAX_TEST_ATTEMPTS,默认 2。
另外端到端也可以记录一下哪些是重试 pass-on-retry 通过的。
There was a problem hiding this comment.
修改:CTest 和端到端模型测试都会区分 pass、pass-on-retry、fail;重试次数通过 MAX_TEST_ATTEMPTS 配置,默认 2。
Summary 增加了总览和重试通过用例列表:
[SUMMARY] Pass / Pass on retry / Fail: X / Y / Z
[SUMMARY] Passed on retry: Y
[PASSED ON RETRY] ...
386c26b to
46b97f0
Compare
- Add configurable MAX_TEST_ATTEMPTS for CTest and model test retries - Classify results as pass, pass-on-retry, and fail in the summary - List retry-passed cases and preserve nonzero exit status on failures - Clean checkpoint save directories before retrying save cases
测例失败时自动重试一次,重试覆盖原日志;再次失败后记录结果并继续执行后续测例。
运行结果示例如下:
