PR preview deploys fail on every PR: the preview host ran out of disk quota and now drops TLS connections #695

The Deploy PR Preview workflow (.github/workflows/pr-preview.yaml) has failed on every PR since 2026-09-25 15:58 UTC. The last success was at 15:11 that day, and about 50 runs across 15 branches have failed since. The Docusaurus build succeeds each time; the Deploy to HarperDB step (harper deploy … project=pr-<N>) is what fails.

What the step reports

• Disk quota exceeded, intermittently from 2026-09-23 and on every run from 2026-09-25: the preview Harper returns a 500 with errno 122 (EDQUOT) while writing the uploaded component, e.g. Unknown system error -122 … open '/home/harperdb/harper/components/pr-685/build/reference/v4/configuration/operations.md' (run 36157691184).
• Since 2026-09-26, the connection fails before anything is uploaded: Client network socket disconnected before secure TLS connection was established, connecting to the preview target on port 9925 (run 36487926705, the latest).

The second failure suggests the preview instance went down, or stopped serving TLS, after its disk filled. That's an inference from the logs; nobody has looked at the host yet.

Likely cause to check

Each PR deploys as its own component (project=pr-<N>, deployed with replicated=true), and the Cleanup PR Preview job drops it (harper drop_component) only when the PR closes. Components left behind by PRs whose cleanup never ran or failed would accumulate until the quota fills.

Impact

Reviewers get no preview link, and every open docs PR shows a failing check. The check isn't required, so PRs still merge, and the job still uploads each PR's build as an artifact (pr-<N>-build) that can be downloaded and served locally.

To fix

• Free space on the preview host, including the components/pr-<N> of closed PRs.
• Restart the preview instance and confirm it serves TLS on 9925.
• Consider pruning closed PRs' components on each deploy, or alerting on the host's disk usage, so a full disk is caught before previews stop.

Fresh full-text documentation preview failures

Two fresh preview jobs passed Build Docusaurus site and uploaded their build artifacts, then failed Deploy to HarperDB with HTTP 500 and Unknown system error -122, write:

• Document full-text paging and configurable search limits: successful 410-document build; preview job evidence.
• Document native full-text search: successful 414-document build; preview job evidence.

These observations confirm a server-side deployment write failure after successful source builds. They do not independently establish quota exhaustion, accumulated preview components, or another root cause. The earlier quota hypothesis above remains a hypothesis for these runs until the owning infrastructure is inspected.

No deployment retry, remote server change or storage cleanup was performed for this observation. Existing issue priority, type, parent, assignment, milestone and other fields are preserved.

Comment generated by kAIle (GPT-6)