Skip to content

Recover nested generator comprehensions through statement parsing - #24

Open
EiNSTeiN- wants to merge 2 commits into
ejfkdev:mainfrom
EiNSTeiN-:fix/nested-generator-comprehensions
Open

EiNSTeiN- wants to merge 2 commits into
ejfkdev:mainfrom
EiNSTeiN-:fix/nested-generator-comprehensions

Conversation

@EiNSTeiN-

Copy link
Copy Markdown

I am opening this PR to keep nested comprehensions inside the value yielded by a generator expression. The code below currently decompiles to invalid source:

def select(rows):
    return tuple({key: row[key] for key in row} for row in rows)

The current output flattens the inner comprehension into the outer generator's clauses:

def select(rows):
    return tuple((key: row[key] for row in rows for key in {}))

The dedicated generator simulator loses which collection is constructed for each yielded row. The new generator_parts_from_body helper instead takes the statement parser's For/If/yield tree and derives the outer generator's clauses while leaving the yielded nested-comprehension expression intact. It restores the actual outer iterable at the call site.

Variations include inner list and set comprehensions, filters, nested comprehensions used as call arguments, and unpacked dictionary targets:

tuple([value * 2 for value in row] for row in rows)
tuple({value * 2 for value in row} for row in rows)
tuple(collect([value * 2 for value in row]) for row in rows)
tuple({key: value for key, value in zip(columns, row) if key}
      for row in rows)

The extractor accepts only a completely recovered generator-expression-shaped tree. Unsupported assignments, loop-else suites and multiple sibling yield paths make extraction fail; they cannot disappear into an apparently valid expression. Structural pass, None-return and constant-expression remnants are tolerated. Other shapes retain the existing fallback.

The new route applies to Python 3.12+ generators containing LOAD_FAST_AND_CLEAR, matching the introduction of PEP 709's inlined comprehensions. The source constructs are older: nested list-comprehension generators begin in Python 2.4; dictionary/set comprehensions are available in Python 3.0 and the Python 2.7 backport.

docs/LIMITATIONS.md removes the repaired minimal nested-comprehension limitation while retaining the full-module verification backlog for the more complex package examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant