Skip to content

fix(python): a parameterized base links its class, like class Leaf(Base[str]) - #2998

Closed
JosefAschauer wants to merge 1 commit into
colbymchenry:mainfrom
JosefAschauer:fix/python-generic-base
Closed

JosefAschauer wants to merge 1 commit into
colbymchenry:mainfrom
JosefAschauer:fix/python-generic-base

Conversation

@JosefAschauer

@JosefAschauer JosefAschauer commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A Python class with a parameterized base, class Leaf(Base[str]), has no extends edge. The grammar parses Base[str] in the base list as a subscript, and the base-list reader only took bare identifiers and attributes, so the base was skipped. Everything that follows the class hierarchy misses such a class. This is the Python counterpart of #2856 (Java/TS Store<String>), and the "generic bases" item left open in #2076.

The resolver also reads a class's base list as text (pythonBases in name-matcher.ts, behind module-global and self. dispatch). It looked up Base[str] verbatim and split Pair[str, int] at its inner comma, so current = Leaf() then current.run() did not reach Base.run.

Fix

  • Extraction, both arms: a subscript base is unwrapped to its value field before the existing identifier | attribute check, in src/extraction/tree-sitter.ts and in the kernel's codegraph-kernel/src/python.rs. The reference sits at the value's start, the position the plain form had.
  • Resolver: pythonBases drops bracketed type arguments (nested ones too) before splitting the list.
  • Not linked, as before: Generic[T] / Protocol[T] (outside the project, like their plain forms), a computed base like make()[0], a subscript of a subscript like Base[int][str], and keyword arguments like metaclass=Meta.

Validation

  • New __tests__/python-generic-bases.test.ts, run on both arms: wasm (CODEGRAPH_KERNEL=0) and kernel (CODEGRAPH_KERNEL_LANGS=python; skipped when no kernel is staged). 16 tests:

    • Base[str], Base[list[int]], Pair[str, dict[str, int]], a base written over several lines, a mixed list (Mixin, Base[str]), and models.Base[int] all link like their plain forms;
    • a keyword argument, a computed base and Base[int][str] link nothing; Generic[T] links nothing;
    • a module global holding Leaf / Two resolves run to Base::run / Pair::run, with a same-named decoy class in the file.

    Each fix was mutation-checked on its own: disabling the wasm unwrap fails only wasm tests, disabling the kernel unwrap fails only kernel tests, and disabling the pythonBases change fails the module-global test on both arms.

  • npx tsc --noEmit: clean. Kernel built with npm run build:kernel.

  • Full suite on Linux with the kernel staged: 9281 passed, 4 failed. The failures are environmental and also fail on main:

    • bundle-launcher ×2: no zip binary.
    • cli-sync ×2: the host runs Node 26, which prints the unsupported-Node banner.
  • Together with feat(resolution): link Python base methods to their overrides (python-override) #2076, Base.run gets its override link to Leaf.run on both arms.

Follow-ups

Independent of #2076.

🤖 Generated with Claude Code

https://claude.ai/code/session_018hJjBUPFiLSFBssshFwHzq

…se[str])

The base-list reader took only bare identifiers and attributes, so a
subscripted base (`Base[str]`, `models.Base[int]`) produced no `extends`
reference and the class had no supertype edge. Both arms (wasm
TreeSitterExtractor and the kernel's python.rs) now name the subscript's
value. Generic[T] and other bases outside the project stay unresolved, as
their plain forms do.

The resolver's own text reader of a class's base list (pythonBases, used by
module-global and self. dispatch) split `Pair[str, int]` at its inner comma
and looked up `Base[str]` verbatim; it now drops bracketed type arguments
first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hJjBUPFiLSFBssshFwHzq
@colbymchenry

Copy link
Copy Markdown
Owner

Thanks for this, and for testing both extraction paths! On current main a parameterized base like class Leaf(Base[str]) gets no extends edge, and a module global holding one reaches nothing. Applied to the newest main, your change fixes both on the wasm path. When you mark it ready, we'll run the kernel arm and an A/B on real Python repos before landing it.

@JosefAschauer

Copy link
Copy Markdown
Contributor Author

Thanks for checking both paths. It's finished from my side; marking it ready for the kernel arm and the A/B.

@JosefAschauer
JosefAschauer marked this pull request as ready for review October 30, 2026 13:59
colbymchenry pushed a commit that referenced this pull request Oct 30, 2026
…se[str])

A Python class with a parameterized base had no extends edge. The grammar
parses Base[str] in a base list as a subscript, and the base-list reader took
only bare identifiers and attributes, so the base was skipped, on the wasm path
and in the kernel alike. Everything that follows the class hierarchy missed
such a class. The resolver's own text reader of a base list, pythonBases, also
looked up Base[str] verbatim and split Pair[str, int] at its inner comma, so
current = Leaf() then current.run() did not reach Base.run.

A subscript base is now unwrapped to its value before the identifier or
attribute check, in src/extraction/tree-sitter.ts and in the kernel's
python.rs. pythonBases drops bracketed type arguments before splitting the list.
Generic[T] and Protocol[T] stay outside the project like their plain forms; a
computed base, a subscript of a subscript and keyword arguments link nothing.

Validation:
- The test file runs on both arms: red on main's source and kernel (12 of 16
  fail, 6 per arm), green here (16 passed).
- Kernel parity with this kernel on 3,155 Python files of flask, requests,
  mypy, litestar, sqlalchemy, polar and django: 0 diffs, and the same 5 files
  defer to wasm as on main. The kernel and wasm edge dumps of the repos run
  both ways are identical.
- A/B against main on 7 Python repos, 457,948 edges: 1,492 added (901 extends,
  555 calls, 36 references), 0 removed, 607 call sites moved, 223 kept their
  target with higher confidence.
  polar: 491 self.get_one_or_none() style calls move from the protocol stub
  RepositoryProtocol at 0.4 to RepositoryBase and its mixin at 0.9.
  litestar: 12 calls in GenericSyncMockRepository move from
  AbstractAsyncRepository to AbstractSyncRepository.
  sqlalchemy: 435 extends and 554 calls added, like t.insert() reaching
  TableClause.insert.
  We read 44 sampled changes and every small bucket. They read right, except
  12 self.op() sites that move to the typing-only overload SQLCoreOperations.op
  and one Enum.copy that moves to TypeEngine.copy.
- 3 of the 901 extends are wrong: sqlalchemy's Sequence[_T], Tuple[int, ...]
  and MutableSet[_T] from typing link to its own classes of those names. The
  plain form does this on main (28 class X(Enum) there), and none of the 3
  brought a call edge.
- Controls express and gin: full dumps byte-identical.
- Index cost with PR #3001 on top, sqlalchemy, median of 3 interleaved: 23.7 s
  on main, 25.2 s here, with 1,143 more edges.

Original PR #2998 by @JosefAschauer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry

Copy link
Copy Markdown
Owner

Thanks @JosefAschauer! This landed as #3101 with you as the commit author, and your fix went in unchanged. The kernel arm and the A/B on real Python repos both came out well: on polar, 491 calls now reach RepositoryBase instead of a protocol stub. We made the test's kernel arm fail rather than skip when a build expects a kernel, and moved the CHANGELOG bullet next to the other Python entries.

colbymchenry pushed a commit that referenced this pull request Oct 30, 2026
Both of the resolver's readers of a Python base list stopped early. pythonBases,
behind module-global and self. dispatch, read only the class line.
declaredSupertypes, behind class-name calls, read three lines and needed the
closing parenthesis on them. black and ruff wrap a long list one base per line,
so class Wrapped(Mixin, Base) written that way seemed to have no bases:
current = Wrapped() then current.run(), and Wrapped.run(x), reached nothing a
base declares. Extraction already read the whole list.

A new pythonBaseList reads from the class line to the list's closing
parenthesis, counting nested parentheses, within the class's extent, and leaves
out comments, so a ) in one doesn't end the list. Both readers use it.
declaredSupertypes still drops type arguments first, the composition the author
posted on PR #3001, and keeps its metaclass= and factory rewrites.

Validation:
- The test file: 7 of 9 fail without this change (the two one-line controls
  pass), 9 pass here. Putting the type-argument drop out of the merged
  case 'python' fails the new composition case and the two of PR #3083.
- Wrapped.run(None) on a wrapped base list reaches Base::run with the kernel
  and with wasm, as do a list over five lines, a decorated class, and a wrapped
  Base[Optional[User]].
- A/B on top of PR #2998, 7 Python repos, 459,440 edges: 118 added (102 calls,
  16 references), 0 removed, 959 call sites moved, 113 kept their target with
  higher confidence. requests, flask, litestar, mypy and django do not change.
  polar: 942 XRepository.from_session(session) calls move from a sibling
  repository's from_session at 0.65 to RepositoryBase.from_session at 0.85.
  sqlalchemy: wrapped heads like class Table( and class ColumnElement( now
  reach what they inherit: t.select() reaches FromClause.select. Its 17 moved
  sites read right; 15 of them land on the typing-only SQLCoreOperations
  overload of the same ColumnOperators method.
- Controls express and gin: full dumps byte-identical.
- No kernel change. Index cost with PR #2998, sqlalchemy, median of 3
  interleaved: 23.7 s on main, 25.2 s here.

Original PR #3001 by @JosefAschauer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
colbymchenry added a commit that referenced this pull request Oct 30, 2026
…t class of that name (#3109)

A Python class whose base the file imports from a package, like
class Color(Enum) after from enum import Enum, linked to any project class
named Enum: the import found no project file, and the name match or a framework
resolver then took the first class of that name. sqlalchemy's enums all
extended its own sql/sqltypes.py Enum, and litestar's pydantic models extended
a BaseModel in a docs example. PR #2998 extended this to parameterized bases,
like Sequence[_T] from typing.

isBoundToOutOfRepoImport is the gate that drops such an edge for JS, TS and
Rust. It now answers for Python too: the name is outside when every import
binding it is absolute and resolveImportPath, which searches for a module the
way Python does, finds none. A class in a file the import path names stays,
because that search can miss a module on a path the program sets up itself, and
so does a class of the referring file, which may rebind the name. A name no
named import binds, like one from a star import, is left as it was.

Validation:
- New python-outside-bases.test.ts: the 7 outside bases fail on main, plain and
  parameterized, and every case that must keep its edge passes both ways.
  Green here (7 passed). Taking out each of the 4 guards fails exactly its own
  test.
- A/B on 7 Python repos, 469,927 edges with the controls: 67 extends removed, 0
  added, 0 moved. All 67 read against their imports and all were wrong:
  sqlalchemy 31 (28 from enum, 3 from typing), litestar 21 (pydantic
  BaseModel), polar 14 (11 fastapi APIRoute, 2 enum IntEnum, 1 opentelemetry
  Meter), django 1 (ctypes Union).
- Controls express and gin unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants