Repository navigation
Expand use of sql parsing string expressions in DataFrame #1278
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This is a follow on to #1273
There are a number of places where it would be convenient to pass SQL strings as expressions. For example, it would be nice to do
df.select(
"a",
"a - b",
col("c"),
)
This should intuitively know that we are getting column a, followed by col("a") - col("b") followed by column 'c'.
Describe the solution you'd like
Using the sql parsing on the DataFrame make the following functions handle SQL strings. We must be very careful that we do not break things like cases where users have a column name that is not SQL parseable.
Describe alternatives you've considered
Status quo
Additional context
DataFrame functions to update:
- select
- remove
select_exprs -
with_columnfeat:with_columnsupports SQL expression #1284 -
with_columnsfeat: Add SQL expression support forwith_columns#1286 - aggregate
-
repartition_by_hashfeat: Add SQL expression forrepartition_by_hash#1285
We do not want to apply this treatment to joins because there is no easy way to know which DataFrame to perform the SQL parsing against.
Activity
timsaucer commentedon Oct 15, 2025
cc @milenkovicm @K-dash
timsaucer commentedon Oct 15, 2025
I think when we have a string we want to first check to see if it is a column name. If so, create a column expression. If not, try sql parsing.
milenkovicm commentedon Oct 15, 2025
Have you changed your mind about select Tim? I'm stuck next few days but may help if needed
timsaucer commentedon Oct 15, 2025
I do think we want
selectbut it just can't be the snippet I shared before. Instead I think it needs to check if the string matches a column name first. If so, make a column expression. Otherwise try parsing. What do you think?milenkovicm commentedon Oct 15, 2025
I have no strong opinion Tim, I'm split between keeping it as it is and change it
milenkovicm commentedon Oct 16, 2025
i hope to tick off few of them off the list this weekend
milenkovicm commentedon Oct 20, 2025
looking at spark it looks like we're aligned with
selectandselectExprmaybe we keep both as they are ?timsaucer commentedon Oct 20, 2025
I can see how it might be preferable to some to have it split out, but I also feel like just having one
selectis very ergonomic. Asking for another opinion from some users I know @ntjohnson1 @MrPowersmilenkovicm commentedon Oct 20, 2025
if we do not have special handle for column names I would agree with you, one select may make more sense. If we have to check for column name before we
parse_sqlI believe it would be simpler from user perspective to leave them as they are. Anyway leats get consensusntjohnson1 commentedon Oct 20, 2025
I definitely think a single
selectwould be nice ergonomically.milenkovicm commentedon Oct 21, 2025
aggregatemay have the same issue likeselectas it expects column name as a string, I'll leave those two to you @timsaucer