Skip to content

Fix unicode issues - #671

Merged
evertlammerts merged 4 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:fix-lone-surrogate-parameter
Oct 16, 2026
Merged

evertlammerts merged 4 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:fix-lone-surrogate-parameter

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

A Python str that UTF-8 cannot encode, such as "\ud800", failed on every value path with
RuntimeError: std::bad_cast: execute("SELECT ?", ["\ud800"]), executemany, sql(params=),
duckdb.Value, lists, dicts and maps, ConstantExpression, values, native UDF return values. In a
pandas or numpy object column it was worse: the scan handed the surrogate to an engine helper, which
raised an INTERNAL error and invalidated the whole database:

df = pd.DataFrame({"s": pd.Series(["\ud800"], dtype=object)})
con.sql("SELECT s FROM df").fetchall()
# INTERNAL Error: invalid code point detected in Utf8Proc::CodepointLength ...
con.sql("SELECT 42")
# FatalException: database has been invalidated

Both happened on 1.5.6 as well (there the first message was pybind11's "Unable to cast Python instance").

Root cause: nanobind's string caster clears the UnicodeEncodeError and reports a failed cast, so the
reason was lost at the first conversion; the object column scan decoded code points itself and never
checked for the surrogate range.

Changes:

  • TransformPythonString converts a str with the same CPython call nanobind uses and, on a
    UnicodeEncodeError, raises InvalidInputException: Python Conversion Failure: cannot encode str as UTF-8, surrogate '\udXXX' at position N is not allowed. Used by the value converter, numpy <U enum
    entries, object category entries, and the read_csv na_values, delimiter and sep and write_csv
    na_rep options. Valid strs take the same path as before.
  • The object column scan (DecodePythonUnicode) computes UTF-8 lengths and encodes inline, raising the
    same error on a surrogate. It no longer calls the engine's per code point helpers, which also makes
    scans of non ASCII object columns without cached UTF-8 about 2x faster (measured on Release builds
    against the base; every other case is within noise).

Not changed: a str used as a name (a dict key becoming a field name, a named parameter name, SQL text)
still fails as before; duckdb.Value still checks nothing at construction. A str subclass such as
numpy.str_ with non ASCII text in an object column fails with an unrelated error, as on 1.5.6.

@evertlammerts
evertlammerts added this pull request to the merge queue Oct 15, 2026
@evertlammerts
evertlammerts removed this pull request from the merge queue due to a manual request Oct 15, 2026
@evertlammerts
evertlammerts force-pushed the fix-lone-surrogate-parameter branch from 5d68906 to 316aa69 Compare October 15, 2026 20:58
@evertlammerts
evertlammerts added this pull request to the merge queue Oct 16, 2026
Merged via the queue into duckdb:v2.0-cyanoptera with commit 6d67c23 Oct 16, 2026
30 checks passed
@evertlammerts
evertlammerts deleted the fix-lone-surrogate-parameter branch October 16, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant