Repository navigation
test/install errors pass silently on PG 9.5-15: pg_regress ignores psql's exit status before PG16 #148
Description
Claude here, filing this at the maintainer's request.
Problem: on PostgreSQL 15 and earlier, a test/install/*.sql file that hits an error with ON_ERROR_STOP set still reports ok, and the error does not fail make test. pgxntool's docs say the opposite. base.mk's test/install section says psql's non-zero exit is something "pg_regress does report as a failure". README.asc's test/install warning ("The only thing that still fails the build is a hard SQL error, and only if the file sets ON_ERROR_STOP") and CLAUDE.md's test/install note say the same. That's only true on PG16 and later.
Why: before PG16, pg_regress decides pass/fail from the output diff alone. A non-zero exit from a test's psql process only appends a note to that test's status line, e.g. ok ... (test process exited with exit code 3). PostgreSQL commit 337903a16fb ("Consider a failed process as a failed test in pg_regress", first released in 16.0) changed this. It was not backpatched; REL_15_STABLE still has the old logic, and 9.5 behaves the same as 15. test/install's expected and actual output are the same file, so its diff can never fail. On PG 15 and earlier, nothing in the install step can fail the run.
Effect: an install-file error that keeps the extension from being installed is still caught on older versions, indirectly, because the test/sql files that follow it then fail. An error raised after the install succeeded is not caught at all, e.g. an assertion at the end of the file such as a version check after ALTER EXTENSION UPDATE, or a check that installing left session settings unchanged. The ON_ERROR_STOP check from #97 makes sure such an error stops psql, but on PG 15 and earlier that stop never reaches the build result. Postgres-Extensions/trunklet#10 and #11 have assertions of exactly this kind in test/install/load.sql.
Possible fixes (not tested):
- After pg_regress runs, fail if its log (
regression.outin the output directory) containsexited with exit codeorterminated by signal. pg_regress writes that note on every supported version, so this would give PG16's behavior to older versions, for every test rather than only test/install. - Alternatively, check each test/install
.outfile for psql error lines. pg_regress runs psql with stderr redirected into the.outfile, so the error text lands there. - Either way, the three doc passages above should say what actually happens on PG 15 and earlier.
Related: #97 (the ON_ERROR_STOP check), #104 (test/install .out cleanup), and open PR #144, which edits the same README text.
Activity
jnasbyupgrade commentedon Nov 2, 2026
Claude here. On PG16+ a failing
test/installscript also doesn't failmake test, so this issue's scope and title should probably cover all PostgreSQL versions, not just 9.5-15. The "That's only true on PG16 and later" assumption doesn't hold fortest/install.Mechanism, on current master (e80485e):
base.mk:438has.IGNORE: installcheck, so pg_regress's exit status never reaches make.test(base.mk:532-533) fails only ifregression.diffsexists.test/installnever produces a diff (see the comment atbase.mk:200). The schedule entry../install/<name>makes the expected and actual output the same file, so pg_regress overwrites it instead of comparing.verify-results(verify-results-pgtap.sh:18) scans only$TESTOUT/results/*.out. The install output is written totest/install/, outside that glob. Anot okfor the install step is printed on pg_regress's stdout and is not in any scanned file either.Observed in Postgres-Extensions/object_reference#45 with a deliberately failing
test/install/load.sqlon PG17:makeexited 0 and pg_regress printednot ok 1 - ../install/load.make verify-resultsalso passed, because the later tests reinstall the extension and pass.makefails only if some later test also fails.On PR #141: its check (
test/bin/check-tap-output.sh, run from thetestrecipe andverify-results) scans thetest/installoutput files, but only for pgTap lines (not ok N, plan problems). A plain-SQL install file whose output has anERROR:and no TAP lines passes that check, so it doesn't fix this case on any version. ItsERROR:scan covers onlyresults/*.out. The pg_regressnot ok/ok (test process exited with exit code N)line is never read. I haven't run #141, so this is from reading the code.