CI: Make the internal LLMEvalKit install diagnosable, and skip it when GHES is unreachable by yoavkatz · Pull Request #1999 · IBM/unitxt

yoavkatz · 2026-11-07T14:48:06Z

.github/actions/install-internal-pip installs the internal MLT/LLMEvalKit package from IBM GHES over SSH. Two problems, both live right now:

When it failed, the entire log said ##[error]Process completed with exit code 1 and nothing else.
When GHES is unreachable, it failed the job — blocking every PR in the repo on an outage no PR can fix.

On 2026-11-07 github.ibm.com:22 became unroutable from GitHub-hosted runners and took down unittests, eager and all ten preparation shards, on main and on every open PR at once. PR #1997 looked like it had broken fourteen jobs; it had broken none.

1. The failure is now legible

GitHub masks this step's output because it carries a secret, and it ran under -e -o pipefail, so the first failing command aborted it silently. Five unrelated causes were indistinguishable. Diagnosing the outage required noticing that an unconditional echo never printed — not a reasonable ask.

Each precondition is now checked separately with its own annotation, visible in the job summary without opening a log:

Annotation	Step	Meaning
`Internal install skipped` (notice)	passes	No secret — fork or Dependabot
`GHES unreachable, install skipped` (warning)	passes	Host did not answer in 15s
`Malformed SSH key`	fails	`ssh-keygen` cannot load the secret
`SSH key rejected`	fails	Host refused the deploy key
`Internal install failed`	fails	Host and key fine; pip failed
`pydantic pin failed`	fails	Package installed, the pin did not

Confirmed in CI on this branch — the annotations API returned level: failure, title: GHES unreachable, and the -T 15 bound worked (the step took 15.2s instead of hanging).

2. An outage no longer fails the job

An unreachable host is now handled like a missing secret: warn, skip the install, let the jobs run.

This is safe because the test suite already has the mechanism — requires_llmevalkit in tests/library/test_metrics.py skips those tests when the package is absent, and its comment already anticipates "CI installs only when repository secrets are available." 15 guarded tests, none unguarded. Imports in src/unitxt/metrics.py are lazy and _requirements_list raises a clear install error, so nothing else is affected. Verified that a guarded test with the package absent skips and leaves the run successful.

A malformed or rejected key still fails, because those are misconfigurations in this repository rather than outages, and skipping them would hide a broken secret indefinitely. This skips what we cannot fix and still fails what we can.

The trade, stated plainly

A green run no longer proves the llmevalkit metrics were exercised. That is the cost of not blocking the repository on a third party, and it is recorded in .github/workflows/README.md rather than left implicit behind a green tick. A GHES unreachable, install skipped warning, or skipped Reflection* tests, is how to tell.

Points worth a reviewer's attention

set -e is dropped deliberately. Every stage reports before it exits, so the step can no longer abort without an explanation. -o pipefail is kept.

The auth probe matches ssh's message, not its exit status. A successful ssh -T git@github.com exits 1 and prints Hi <user>! You've successfully authenticated — keying on exit status would fail CI on a good key. Verified in both directions. It also pins to the deploy key with -i/IdentitiesOnly=yes: without that, an agent-offered key answers instead and a rejection is masked. That was a real bug caught in testing.

ssh-keyscan fails silently on an unroutable host (non-zero, no output), so diagnostic echoes are [ -n ... ]-guarded, and the success echo is followed by true so an empty greeting cannot leave a passing step non-zero.

No fingerprint is logged. An earlier revision printed ssh-keygen -l output on success. A fingerprint hashes only the public half and is not a credential, but this repository and its logs are public, so nothing key-derived is emitted at all. The validation is unchanged and still catches malformed and truncated keys.

Inputs reach the script via env: instead of ${{ }} interpolated into the shell body, which also removes a script-injection surface.

Two messages are worded to avoid reading as key: <value>, which detect-secrets flags. Reworded rather than marked pragma: allowlist secret, so the hook keeps working on this file.

Compatibility

No change to the action's inputs or defaults. A working install behaves exactly as before. All four call sites use defaults only and are untouched.

Verification

All five paths exercised locally against a real SSH endpoint with a stubbed pip: success, no-secret skip, unreachable host (now exit 0 + warning), malformed/truncated key (exit 1), rejected key (exit 1), pip failure (exit 1). YAML parses, bash -n clean, pre-commit clean.

GHES-specific paths cannot be reproduced outside the IBM network, so github.com stood in for the host: what is verified is the classification logic, not GHES itself.

Still outstanding

This makes the outage survivable, not fixed. Someone with IBM GHES/network access still needs to restore egress to github.ibm.com:22 before those metrics are actually tested again. LLMEVALKIT_SSH_KEY is confirmed not a factor — ssh-keygen loads it fine, so the earlier suspicion about its age can be dropped.

🤖 Generated with Claude Code

A failure in `.github/actions/install-internal-pip` was almost unreadable. GitHub masks the step's output because it carries a secret, and the step ran under `-e -o pipefail`, so any failing command aborted it with no output at all: ##[error]Process completed with exit code 1 Five distinct causes -- absent secret, malformed secret, unreachable GHES, rejected credential, broken package -- all looked exactly alike. This is not hypothetical: it is how CI is failing right now. `unittests`, `eager`, `performance` and all ten `preparation` shards die in this step, on main as well as on every open PR, and the only way to tell that the cause is network reachability rather than the code under test is to notice that the unconditional `echo "Installing from URL: ..."` never printed, so execution must have stopped in the three commands before it. Each precondition is now checked separately and fails with its own `::error` annotation, which shows in the job summary without opening the log: - Malformed SSH key -- `ssh-keygen` cannot load the secret - GHES unreachable -- `ssh-keyscan -T 15` got no host key - SSH key rejected -- the host refused the deploy key - Internal install failed -- host and credential fine, pip failed - pydantic pin failed -- the package installed, the pin did not Each message says which external dependency broke and where to look, and the two infrastructure cases say outright that the pull request is not at fault. Implementation notes: `set -e` is dropped deliberately. Every stage now reports before it exits, so the step can no longer abort without an explanation; `-o pipefail` is kept. The auth probe matches ssh's *message*, not its exit status: a successful `ssh -T git@github.com` exits 1 and prints "Hi <user>! You've successfully authenticated". Keying on the status would fail CI on a perfectly good key. It also passes `-i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes`, so an agent key cannot answer in place of the deploy key and mask a rejection. `ssh-keyscan` exits non-zero but silently on an unroutable host, so each diagnostic echo is guarded with `[ -n ... ]` rather than printing an empty line, and the success-path echo is followed by `true` so an empty greeting cannot leave a passing step with a non-zero status. Inputs now reach the script through `env:` instead of being interpolated into the shell body with `${{ }}`, which also removes a script-injection surface. The "malformed key" message is worded to avoid reading as `key: <value>`, which `detect-secrets` flags as a Secret Keyword. Reworded rather than marked `pragma: allowlist secret`, so the hook keeps working on this file. No change to the action's inputs, its defaults, or its behaviour on success: a working install prints the same progress and exits 0, and a run with no secret still skips with a notice and passes. The key check prints only the type and fingerprint, never anything derived from the key material. All six paths were exercised locally against a real SSH endpoint with a stubbed pip -- success, skip-on-no-secret, malformed key, unreachable host, rejected key and pip failure -- each confirmed to give the intended annotation and exit status. The GHES-specific paths cannot be reproduced outside the IBM network, so `github.com` stood in for the host; what is verified is the classification logic, not GHES itself. This only improves the error message. It does not fix the outage: the install will keep failing until `github.ibm.com` is reachable from GitHub-hosted runners again, but it will now say so. Signed-off-by: Yoav Katz <katz@il.ibm.com>

The key check logged `ssh-keygen -l` output on success, which named the deploy key's fingerprint, bit size and comment: Deploy key loaded: 256 SHA256:<...> gha-llmevalkit (ED25519) A fingerprint is a hash of the *public* half -- `ssh-keygen -l` returns the same value for a key and its `.pub`, and it can be recomputed from the public blob alone -- so it is not a credential and cannot be used to authenticate. But this repository is public and so are its CI logs, and nothing here needs the key to appear in them, so it no longer does. The validation itself is unchanged and still catches a truncated or wrongly-wrapped secret before it surfaces as an opaque "Permission denied": `ssh-keygen -y` still parses the key, its output is now discarded, and the success line just says the key is well-formed. Verified that a good key is still accepted and that both a malformed and a truncated key are still detected. What is lost is the ability to tell from the log *which* key is deployed, so a wrong-but-valid key now looks the same as a revoked one: both reach the `SSH key rejected` annotation. That is the intended trade. No other output was key-derived. The remaining echoes carry ssh's own messages ("Permission denied (publickey)", the authenticated greeting), the host, the repo and the pip URL -- checked for fingerprints, comments and key material, none present. Signed-off-by: Yoav Katz <katz@il.ibm.com>

An unreachable `github.ibm.com` is now a warning and a skip, not a failure. A GHES outage is someone else's infrastructure, and failing the job on it blocks every pull request in the repository on something no pull request can fix. That is not hypothetical: on 2026-11-07 `github.ibm.com:22` became unroutable from GitHub-hosted runners and took down `unittests`, `eager` and all ten `preparation` shards -- on `main` and on every open PR at once, with each PR appearing to have broken fourteen jobs. This handles an unreachable host the way the action already handles a missing secret: say so, skip the install, let the jobs run. The tests that need the package already skip themselves when it is absent, via `requires_llmevalkit` in `tests/library/test_metrics.py` -- 15 guarded tests, and none unguarded -- so the suite stays green and simply covers less. The imports in `src/unitxt/metrics.py` are lazy and `_requirements_list` already raises a clear install error, so nothing else is affected. A malformed or rejected key still fails. Those are misconfigurations in this repository, not outages, and skipping them silently would hide a broken secret indefinitely. The distinction is the point: this skips what we cannot fix and still fails what we can. The trade is deliberate, and recorded in the workflows README because it is not self-evident from a green tick: a passing run no longer proves the `llmevalkit` metrics were exercised. A `GHES unreachable, install skipped` warning on the job, or skipped `Reflection*` tests in the output, is how to tell. One comment is worded to avoid reading as `key: <value>`, which `detect-secrets` flags as a Secret Keyword, for the same reason as the error message in the previous commit. Verified all five paths locally with a stubbed pip: unreachable host now exits 0 with the warning; malformed key and rejected key still exit 1 with their annotations; success still installs and exits 0; a run with no secret still skips and passes. Also confirmed that a guarded test with the package absent skips rather than fails, leaving the run successful. Signed-off-by: Yoav Katz <katz@il.ibm.com>

yoavkatz added 3 commits November 7, 2026 16:47

yoavkatz changed the title ~~CI: Make the internal LLMEvalKit install step diagnosable~~ CI: Make the internal LLMEvalKit install diagnosable, and skip it when GHES is unreachable Nov 7, 2026

yoavkatz merged commit 196c222 into main Nov 7, 2026
15 of 23 checks passed

yoavkatz deleted the ci/diagnosable-internal-pip branch November 7, 2026 15:23

Navigation Menu

Repository navigation

CI: Make the internal LLMEvalKit install diagnosable, and skip it when GHES is unreachable - #1999

CI: Make the internal LLMEvalKit install diagnosable, and skip it when GHES is unreachable#1999
yoavkatz merged 3 commits into
mainfrom
ci/diagnosable-internal-pip

yoavkatz commented Nov 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

yoavkatz commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. The failure is now legible

2. An outage no longer fails the job

The trade, stated plainly

Points worth a reviewer's attention

Compatibility

Verification

Still outstanding

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

yoavkatz commented Nov 7, 2026 •

edited

Loading