Repository navigation

Fix PySpark struct() to accept a list of columns - #378

Closed

wavebyrd wants to merge 612 commits into

duckdb:mainfrom

wavebyrd:fix/17189-spark-struct-list

Closed

Fix PySpark struct() to accept a list of columns#378
wavebyrd wants to merge 612 commits into
duckdb:mainfrom
wavebyrd:fix/17189-spark-struct-list

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 18, 2025
- Run pre-commit on-push
  evertlammerts
  committed
  3af6f8f
- Very subtle typing problem in the tests
  evertlammerts
  committed
  5077501
- Merge branch 'main' into readme-update
  evertlammerts
  authored
  fad8ab9
- docs: update readme and move contributing docs to CONTRIBUTING.md (#55)
  evertlammerts
  authored
  d8e2a54
- Add code quality workflow
  evertlammerts
  committed
  30966ec
- fix sdist build
  evertlammerts
  committed
  6d7684c
Commits on Sep 22, 2025
- Add linting and formatting (#72)
  evertlammerts
  authored
  b83bdf8
- re-added adbc_driver_duckdb to exported packages
  evertlammerts
  committed
  40c19e6
Commits on Sep 23, 2025
- Merge branch 'v1.4-andium' into reset_pyerr_before_throwing
  evertlammerts
  authored
  3f73855
- Fixed adbc driver and force tests to run
  evertlammerts
  committed
  68b7702
- Reset PyErr before throwing (#69)
  evertlammerts
  authored
  d0fb192
- Cleaned up
  evertlammerts
  committed
  091b587
Commits on Sep 24, 2025
- double extern?
  evertlammerts
  committed
  ec5160e
- try to force inclusion with msvc and gcc as well
  evertlammerts
  committed
  956430b
- Merge branch 'v1.4-andium' into adbc_driver
  evertlammerts
  authored
  38f86e7
Commits on Sep 25, 2025
- pinning adbc-driver-manager to 1.7.0
  evertlammerts
  committed
  430cc11
- xfail adbc tests on windows
  evertlammerts
  committed
  2ff5448
- Fix adbc driver (#81)
  evertlammerts
  authored
  5ae970d
- bump submodule
  evertlammerts
  committed
  642c57a
- bump submodule (#86)
  evertlammerts
  authored
  6e115be
- CI: Add jobs to mirror issues
  szarnyasg
  committed
  d13e971
Commits on Sep 26, 2025
- bump submodule
  evertlammerts
  committed
  4a6c4cb
- skip failing test while bug is fixed in core
  evertlammerts
  committed
  d365112
- now xfail correct test :/
  evertlammerts
  authored
  20848e8
- Merge branch 'main' into test_skip_19154
  evertlammerts
  authored
  fd69319
- Skip test to get working build (#89)
  evertlammerts
  authored
  0e120c9
- CI: Fix repository name
  szarnyasg
  committed
  24254c8
Commits on Oct 1, 2025
- bumped submodule
  evertlammerts
  committed
  132871e
- added stubs and mypy
  evertlammerts
  committed
  5ad0ffd
- mypy passing
  evertlammerts
  committed
  8d2faa4
- make sure all constants are in
  evertlammerts
  committed
  09d3ae0
- fixed tests
  evertlammerts
  committed
  7904d7e
- added missing overloads
  evertlammerts
  committed
  5385777
Commits on Oct 2, 2025
- fixed stub mypy errors
  evertlammerts
  committed
  b437aa2
- deprecate typing and functional
  evertlammerts
  committed
  62f8c8d
- add stubs to sdist
  evertlammerts
  committed
  4c00ddc
- CI: Add jobs to mirror issues (#88)
  evertlammerts
  authored
  0564204
- Adding stubs and making mypy pass (#96)
  evertlammerts
  authored
  ef0492c
- Bump submodule
  evertlammerts
  committed
  d51e60b
Commits on Oct 3, 2025
- disable lazy pushdown with hugeint test for now
  evertlammerts
  committed
  e0a3f3b
Commits on Oct 4, 2025
- patch hugeint to polars decimal
  evertlammerts
  committed
  2c96769
- tests
  evertlammerts
  committed
  b7aaad3
- tests
  evertlammerts
  committed
  33f04be
- Fix Polars expr pushdown (#102)
  evertlammerts
  authored
  da92cb8
Commits on Oct 6, 2025
- bump duckdb submodule to 1.4.1 sha
  evertlammerts
  committed
  1cd589c
- pre-commit mypy config
  evertlammerts
  committed
  45d1016
- pre-commit mypy config (#106)
  evertlammerts
  authored
  cdaa708
- fix versioning
  evertlammerts
  committed
  9841600
- no need to check for dirty versions
  evertlammerts
  committed
  688a682
Commits on Oct 7, 2025
- added changelog
  evertlammerts
  committed
  01ef21d
- Merge branch 'v1.4-andium'
  evertlammerts
  committed
  dd83171
- fixed linter warnings
  evertlammerts
  committed
  55d2384
- fix arrow import issues in tests for py314
  evertlammerts
  committed
  e50375f
- Merge branch 'v1.4-andium' (#109)
  evertlammerts
  authored
  999f04a
- remove py314t in workflows
  evertlammerts
  committed
  e5b4795
- remove py314t in workflows (#110)
  evertlammerts
  authored
  454b067
Commits on Oct 8, 2025
- README.md and CONTRIBUTING.md equal to main's
  evertlammerts
  committed
  9dc5afb
- Devenv instructions linked
  evertlammerts
  committed
  aaeb20b
- Devenv instructions linked
  evertlammerts
  committed
  5e32e10
- more pre-commit
  evertlammerts
  committed
  995c0da
- more pre-commit
  evertlammerts
  committed
  ce73ce9
- fix submodule post-checkout hook
  evertlammerts
  committed
  adc60f6
- fix submodule post-checkout hook
  evertlammerts
  committed
  6710c60
- bumped submodule
  evertlammerts
  committed
  22de71f
- 3.14 support
  evertlammerts
  committed
  6e33504
- cibuildwheel bump
  evertlammerts
  committed
  231b583
- fix pandas tests
  evertlammerts
  committed
  a752290
Commits on Oct 9, 2025
- disable randomly
  evertlammerts
  committed
  b2cb372
- fix tests
  evertlammerts
  committed
  de186cb
- make sure config is loaded
  evertlammerts
  committed
  8e14670
- replace use of deprecated typing._UnionGenericAlias
  evertlammerts
  committed
  f07aa96
- equality instead of instanceof
  evertlammerts
  committed
  b6c2bea
- fix pytest collect hook
  evertlammerts
  committed
  1844e05
- try to get pytest config work correct for windows runners
  evertlammerts
  committed
  5bd8739
- another try to appease the windows runners
  evertlammerts
  committed
  18556ba
- without PWD
  evertlammerts
  committed
  4fb9474
- trying confcutdir
  evertlammerts
  committed
  1c84810
Commits on Oct 10, 2025
- remove pytest plugins
  evertlammerts
  committed
  4de4296
- Add Python 3.14 support (#116)
  evertlammerts
  authored
  36d2ee1
- move code to the dedicated repository
  Tishj
  committed
  b05c6a9
- added 3.14 to classifiers
  evertlammerts
  committed
  38a2ee8
- Move code to the dedicated repository (#117)
  evertlammerts
  authored
  8369a1b
- bumped submodule
  evertlammerts
  committed
  297e9c8
- bumped submodule
  evertlammerts
  committed
  e47e16f
Commits on Oct 13, 2025
- Fix pypi cleanup script
  evertlammerts
  committed
  26a462c
- Prefer non-verbose mypy output in pre-commit
  evertlammerts
  committed
  7c06527
Commits on Oct 16, 2025
- Remove mypy as test dependency because of install problems in CI
  evertlammerts
  committed
  ba714a2
- Small rework of the pypi cleanup script
  evertlammerts
  committed
  b22266e
- Merge branch 'v1.4-andium'
  evertlammerts
  committed
  7481d34
Commits on Oct 17, 2025
- Link MSVC runtime library statically
  staticlibs
  committed
  71cf7e6
Commits on Oct 28, 2025
- explicitly import importlib.util; remove erroneous print
  Henry Harbeck
  committed
  ef1ace4
- Fix ADBC driver path resolution when `importlib.util` was not implicitly loaded (#135)
  evertlammerts
  authored
  8121bf8
Commits on Oct 29, 2025
- Moving staging to CF
  hannes
  committed
  39dc999
- bumped submodule
  evertlammerts
  committed
  e549a85
- Merge branch 'main' into stagingmove
  hannes
  authored
  5944e9d
- Moving staging to CF (#144)
  evertlammerts
  authored
  8e0b3f7
Commits on Oct 31, 2025
- bumped submodule
  evertlammerts
  committed
  5f81c03
- add targeted test workflow
  evertlammerts
  committed
  3a78e2f
- add targeted test workflow (#145)
  evertlammerts
  authored
  37fe8d2
- explicitly import importlib.util; remove erroneous print
  
  Henry Harbeck
  authored and
  evertlammerts
  committed
  8ca82c5
- bumped submodule
  evertlammerts
  committed
  3750fc0
- add targeted test workflow
  evertlammerts
  committed
  b460c07
- Merge with 1.4-andium (#146)
  evertlammerts
  authored
  c82410a
- Remove xfail annotations on adbc tests
  evertlammerts
  committed
  d7dae8b
- Remove xfail annotations on adbc tests (#147)
  evertlammerts
  authored
  6f8cf08
- fix config dict value typehint
  evertlammerts
  committed
  19db033
- fix config dict value typehint (#151)
  evertlammerts
  authored
  c91f2ab
- Add df data and tz type columns back into the same loc after type conversion
  evertlammerts
  committed
  8503893
- Add df data and tz type columns back into the same loc after type con… (#150)
  evertlammerts
  authored
  8f17f4c
- Enable pyarrow with python 3.14
  evertlammerts
  committed
  e1717d3
Commits on Nov 1, 2025
- Enable pyarrow with python 3.14 (#152)
  evertlammerts
  authored
  d7de884
- use macos-15-intel now that macos-13 is closing down
  evertlammerts
  committed
  d83c28f
- release s3 upload fix
  evertlammerts
  committed
  a589315
Commits on Nov 4, 2025
- spark imports
  evertlammerts
  committed
  2393b7f
- spark imports (#157)
  evertlammerts
  authored
  9ecb3b5
Commits on Nov 5, 2025
- Bumped submodule
  evertlammerts
  committed
  1a5d43b
- Fix failing test due to changed error msg
  evertlammerts
  committed
  8d1371d
- Fix failing test due to changed error msg (#158)
  evertlammerts
  authored
  022bedf
- mypy shouldn't check experimental.spark
  evertlammerts
  committed
  3ffd3f1
- remove experimental import because of the transitive dependencies that requires
  evertlammerts
  committed
  beb6d03
Commits on Nov 7, 2025
- Bumped submodule
  evertlammerts
  committed
  2e1d973
- Bumped submodule
  evertlammerts
  committed
  49f7575
- Fix InsertRelation on attached database
  evertlammerts
  committed
  22d40e9
Commits on Nov 9, 2025
- Add explicit polars overloads
  J-Meyers
  committed
  077c204
Commits on Nov 10, 2025
- review feedback
  evertlammerts
  committed
  ab4a8d4
- Add explicit .pl(lazy=True) overload (#172)
  evertlammerts
  authored
  d3943e8
- Bumped submodule
  evertlammerts
  committed
  18a9549
- Fix InsertRelation on attached database (#155)
  evertlammerts
  authored
  db196c2
Commits on Nov 11, 2025
- Submodule at 68d7555 for 1.4.2 release
  evertlammerts
  committed
  b86a435
Commits on Nov 12, 2025
- Fix project metadata
  evertlammerts
  committed
  5f7c8b1
- Fix project metadata (#174)
  evertlammerts
  authored
  9554f4e
- [minor] add suport for types and print schema
  Dharin-shah
  committed
  dc87730
- fix typo
  Dharin-shah
  committed
  0ca2189
Commits on Nov 13, 2025
- add treeString support for printSchema
  Dharin-shah
  committed
  f3a78d7
- Create a PR to bump submodule if nightly is stale
  evertlammerts
  committed
  7c5a369
Commits on Nov 14, 2025
- fix formatting
  Dharin-shah
  committed
  c03d38e
Commits on Nov 15, 2025
- fix ruff check
  Dharin-shah
  committed
  8f4166c
- fix test
  Dharin-shah
  committed
  0537dc6
Commits on Nov 17, 2025
- fix precommit
  Dharin-shah
  committed
  54ea3b9
Commits on Nov 19, 2025
- Create a PR to bump submodule if nightly is stale (#159)
  evertlammerts
  authored
  389911f
- Bump submodule
  duckdblabs-bot
  committed
  f76cee1
- [duckdb-labs bot] Bump DuckDB submodule (#185)
  evertlammerts
  authored
  24a548d
Commits on Nov 20, 2025
- Bump submodule
  duckdblabs-bot
  committed
  368297e
- [duckdb-labs bot] Bump DuckDB submodule (#186)
  evertlammerts
  authored
  6855e38
- [minor][spark] add suport for types and print schema (#175)
  evertlammerts
  authored
  77825a7
Commits on Nov 21, 2025
- Bump submodule
  duckdblabs-bot
  committed
  9860275
- [duckdb-labs bot] Bump DuckDB submodule (#188)
  evertlammerts
  authored
  e8eaa7c
Commits on Nov 24, 2025
- don't fail if the submodule is at the same commit as the input
  evertlammerts
  committed
  4d0c182
Commits on Nov 26, 2025
- Bump submodule
  duckdblabs-bot
  committed
  7979e99
- [duckdb-labs bot] Bump DuckDB submodule (#191)
  evertlammerts
  authored
  2e83154
Commits on Nov 29, 2025
- Bump submodule
  duckdblabs-bot
  committed
  34170f4
- [duckdb-labs bot] Bump DuckDB submodule (#197)
  evertlammerts
  authored
  b09d165
Commits on Nov 30, 2025
- Bump submodule
  duckdblabs-bot
  committed
  2bbbf0a
- [duckdb-labs bot] Bump DuckDB submodule (#198)
  evertlammerts
  authored
  6b63f0a
Commits on Dec 2, 2025
- Bump submodule
  duckdblabs-bot
  committed
  8c37c83
- [duckdb-labs bot] Bump DuckDB submodule (#202)
  evertlammerts
  authored
  82a9a58
- Add filename_pattern to to_parquet Python API
  
  matthew-bayer
  authored and
  evertlammerts
  committed
  a1ba683
- Fix linting and test
  evertlammerts
  committed
  6984fe9
Commits on Dec 3, 2025
- Bump submodule
  duckdblabs-bot
  committed
  bff3399
- Add filename_pattern to to_parquet Python API (#201)
  evertlammerts
  authored
  ee1b539
- [duckdb-labs bot] Bump DuckDB submodule (#203)
  evertlammerts
  authored
  64cae63
Commits on Dec 4, 2025
- Bump submodule
  duckdblabs-bot
  committed
  12afe3c
- [duckdb-labs bot] Bump DuckDB submodule (#206)
  evertlammerts
  authored
  988060d
Commits on Dec 5, 2025
- Bump submodule
  duckdblabs-bot
  committed
  5d4c00d
- [duckdb-labs bot] Bump DuckDB submodule (#207)
  evertlammerts
  authored
  03305b1
- add windows arm64 build
  evertlammerts
  committed
  4a4e62e
- Fix Windows ARM64 excludes in workflow
  evertlammerts
  authored
  87b4999
Commits on Dec 6, 2025
- Bump submodule
  duckdblabs-bot
  committed
  b657b6e
- [duckdb-labs bot] Bump DuckDB submodule (#212)
  evertlammerts
  authored
  aa5ddc6
- Merge branch 'v1.4-andium' into windows_arm64
  evertlammerts
  authored
  a68c65a
- add windows arm64 build (#211)
  evertlammerts
  authored
  471208d
- Merge remote-tracking branch 'upstream/v1.4-andium'
  evertlammerts
  committed
  c3e4947
Commits on Dec 8, 2025
- fic adbc test imports
  evertlammerts
  committed
  7549b30
- fix adbc test imports (#215)
  evertlammerts
  authored
  2af387e
- fix main builds
  evertlammerts
  committed
  96f7f87
- [minor] add suport for types and print schema
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  82f364e
- fix typo
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  4455f65
- add treeString support for printSchema
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  1ac4a42
- fix formatting
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  501635b
- fix NaN response
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  b1071cd
- minor fixes
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  9ccada5
- fix ruff check
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  a301e9f
- fix test
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  cc7e120
- fix precommit
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  64156d9
- revert unncessary change
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  47e87c0
- remove unncessary comment
  
  Dharin-shah
  authored and
  evertlammerts
  committed
  9dfc344
- Merge branch 'v1.4-andium' into minor-bugfixes
  evertlammerts
  authored
  058bddf
- feat: add file_size_bytes to to_parquet
  nicornk
  committed
  eb1c79e
- add file_size_bytes to to_parquet (#204)
  evertlammerts
  authored
  5886923
- [minor][spark] Minor bugfixes (#180)
  evertlammerts
  authored
  b32822c
- pin new duckdb core SHA
  evertlammerts
  committed
  0f021eb
- Merge v1.4-andium and fix build (#214)
  evertlammerts
  authored
  49ae9be
Commits on Dec 10, 2025
- Bump submodule
  evertlammerts
  committed
  8787b2b
Commits on Dec 11, 2025
- bumped submodule
  evertlammerts
  committed
  12e94bc
- fix polars tests
  evertlammerts
  committed
  465b88b
- fix polars tests (#218)
  evertlammerts
  authored
  f6bec20
Commits on Dec 12, 2025
- Bump submodule
  duckdblabs-bot
  committed
  417eb81
- tests for string and binary views
  evertlammerts
  committed
  7bb2447
- [duckdb-labs bot] Bump DuckDB submodule (#220)
  evertlammerts
  authored
  b3515b3
- Merge branch 'v1.4-andium' into arrow_view_filters
  evertlammerts
  authored
  a6ad57f
- Quote view names in unregister
  evertlammerts
  committed
  c5a1de9
Commits on Dec 13, 2025
- Bump submodule
  duckdblabs-bot
  committed
  1906f2e
- [duckdb-labs bot] Bump DuckDB submodule (#223)
  evertlammerts
  authored
  2565a7f
- tests for string and binary views (#221)
  evertlammerts
  authored
  f8c0fbd
Commits on Dec 15, 2025
- Limit string nodes in Polars expressions to constant expressions
  evertlammerts
  committed
  6d54c35
- Quote view names in unregister (#222)
  evertlammerts
  authored
  1d91f8e
- Merge branch 'v1.4-andium' into polars_io_escaping
  evertlammerts
  authored
  3d0d606
- Replace CORE_EXTENSIONS with BUILD_EXTENSIONS
  evertlammerts
  committed
  d561b87
- Explicitly linking extensions
  evertlammerts
  committed
  2331353
- Limit string nodes in Polars expressions to constant expressions (#225)
  evertlammerts
  authored
  9e42f0b
- Merge remote-tracking branch 'origin/v1.4-andium'
  evertlammerts
  committed
  44637e1
Commits on Dec 23, 2025
- Bump submodule
  duckdblabs-bot
  committed
  a73682e
- [duckdb-labs bot] Bump DuckDB submodule (#234)
  evertlammerts
  authored
  379a7fd
Commits on Dec 25, 2025
- Bump submodule
  duckdblabs-bot
  committed
  b4efee4
- [duckdb-labs bot] Bump DuckDB submodule (#237)
  evertlammerts
  authored
  c95b286
Commits on Dec 30, 2025
- Bump submodule
  duckdblabs-bot
  committed
  85f45aa
Commits on Dec 31, 2025
- Fix select_dtypes with quoted identifiers
  Schwarf
  committed
  ce52cd2
Commits on Jan 1, 2026
- Add test to verify bug is fixed.
  Schwarf
  committed
  40e2861
- ci: trigger
  Schwarf
  committed
  2006d2b
- Add new line at EOF.
  Schwarf
  committed
  253f500
- [duckdb-labs bot] Bump DuckDB submodule (#240)
  evertlammerts
  authored
  c1edb5e
Commits on Jan 3, 2026
Commits on Jan 5, 2026
- Fix adbc tests
  evertlammerts
  committed
  9d94f7b
- Fix adbc tests (#248)
  evertlammerts
  authored
  8aa0364
- Merge branch 'main' into fix/select-dtypes-identifier-quoting
  evertlammerts
  authored
  8b1a241
- Exclude jemalloc from anything apart from debug builds on osx and linux
  evertlammerts
  committed
  72bb5c0
- Merge branch 'main' into fix/python-udf-refcount-leak
  evertlammerts
  authored
  634e7a2
- Fix review finding. Use KeywordHelper::WriteOptionallyQuoted
  Schwarf
  committed
  63c0b82
- Exclude jemalloc from anything apart from debug builds on osx and linux
  evertlammerts
  committed
  218d85e
- Fix select_dtypes for quoted column identifiers (#242)
  evertlammerts
  authored
  ca07f26
- Fix jemalloc cmake filter (#249)
  evertlammerts
  authored
  d508e84
- Merge branch 'main' into fix/python-udf-refcount-leak
  evertlammerts
  authored
  7dd6c0e
Commits on Jan 6, 2026
- remove jemalloc extension
  evertlammerts
  committed
  345b3b3
- remove jemalloc extension (#250)
  evertlammerts
  authored
  2278faa
- Merge branch 'main' into msvc_static
  evertlammerts
  authored
  61b2fb2
- Link MSVC runtime library statically (#130)
  evertlammerts
  authored
  057d44d
- enable jemalloc but only on linux 64bit, aligned with duckdb core
  evertlammerts
  committed
  edcbda3
- Bumped submodule
  evertlammerts
  committed
  ad1f013
- fixed linux jemalloc check
  evertlammerts
  committed
  faf18cc
- Fix : aggregate method typing to accept list of expressions
  
  Narwhal-fish
  authored and
  evertlammerts
  committed
  1299b00
- Merge branch 'main' into fix-DuckDBPyRelation.aggregate-incorrect-typing
  evertlammerts
  authored
  46637bc
- Fix : aggregate method typing to accept list of expressions (#246)
  evertlammerts
  authored
  5f80ba1
- fix millisecond typo
  evertlammerts
  committed
  f0ea9df
- Merge branch 'main' into fix/python-udf-refcount-leak
  evertlammerts
  authored
  bab10f7
- Fix typo in class name: milisecond -> millisecond (#252)
  evertlammerts
  authored
  2f30f1a
- Fix Python scalar UDF refcount leak (#245)
  evertlammerts
  authored
  5ab13de
Commits on Jan 12, 2026
- disable release on main
  evertlammerts
  committed
  5686cd1
Commits on Mar 12, 2026
- Fix PySpark struct() to accept a list of columns (#17189)
  wavebyrd
  committed
  1f211e2

Commits

Commits on Sep 18, 2025

Commits on Sep 22, 2025

Commits on Sep 23, 2025

Commits on Sep 24, 2025

Commits on Sep 25, 2025

Commits on Sep 26, 2025

Commits on Oct 1, 2025

Commits on Oct 2, 2025

Commits on Oct 3, 2025

Commits on Oct 4, 2025

Commits on Oct 6, 2025

Commits on Oct 7, 2025

Commits on Oct 8, 2025

Commits on Oct 9, 2025

Commits on Oct 10, 2025

Commits on Oct 13, 2025

Commits on Oct 16, 2025

Commits on Oct 17, 2025

Commits on Oct 28, 2025

Commits on Oct 29, 2025

Commits on Oct 31, 2025

Commits on Nov 1, 2025

Commits on Nov 4, 2025

Commits on Nov 5, 2025

Commits on Nov 7, 2025