moosetechnology · jecisc · Sep 29, 2026 · Sep 18, 2026 · Sep 18, 2026 · Sep 23, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,100 @@
+# AGENTS.md
+
+Always read and follow the coding conventions in `~/.opencode/AGENTS.md` if present.
+
+## What This Is
+
+A **Pharo Smalltalk** project (not Python) that provides a Famix AST representation for Python code, built on TreeSitter. Part of the [moosetechnology/FAST](https://github.com/moosetechnology/FAST) ecosystem.
+
+Source code format: **Tonel** (all source in `src/`).
+
+## Build & Test
+
+This is a Pharo Smalltalk project. There are no npm/pip/make commands.
+
+**Install dependencies in a Pharo image:**
+```smalltalk
+Metacello new
+  githubUser: 'moosetechnology' project: 'FAST-Python' commitish: 'main' path: 'src';
+  baseline: 'FASTPython';
+  load
+```
+
+**CI command (what actually runs):**
+```bash
+smalltalkci -s Pharo64-13   # or Pharo64-14
+```
+CI is defined in `.github/workflows/tests.yml`. Tests run via `smalltalkci` using the `smalltalk.ston` spec. Coverage is collected in lcov format for `FAST-Python.*` packages.
+
+There is no local test runner script outside of `smalltalkci`. You need a Pharo image with the project loaded to run tests interactively.
+
+## Package Structure
+
+```
+src/
+  BaselineOfFASTPython/          -- Metacello baseline (dependency & group definitions)
+  FAST-Python-Model-Generator/   -- Metamodel generator (Famix code generation)
+  FAST-Python-Model/             -- Generated model classes (FASTPy*), do NOT edit by hand
+  FAST-Python-Tools/             -- Importer, CFG, local resolver, SSA, TreeSitter visitor, variable-analysis extensions
+  FAST-Python-Tools-Tests/       -- All tests (tags: Core = abstract test case, Model = model tests, Analysis = variable-analysis tests)
+```
+
+**Baseline groups:**
+- `Core` = Model + Tools
+- `Generator` = Model-Generator only
+- `Tests` = Tools-Tests only (the former Model-Tests package was merged into Tools-Tests)
+
+**Dependencies** (from baseline):
+- `FAST` v3: `github://moosetechnology/FAST:v3/src` (loads `'All'` group)
+- `TreeSitter` v2.0.0: `github://Evref-BL/Pharo-Tree-Sitter:v.2.0.0/src`
+
+## Critical: Model Files Are Generated
+
+`FAST-Python-Model/` contains ~130 generated class files (`FASTPy*`). These are produced by `FASTPythonMetamodelGenerator` in `FAST-Python-Model-Generator/`. **Do not edit model files directly** -- edit the generator and re-run it. The generator also produces a visitor trait (`FASTPyTVisitor`).
+
+## Key Entry Points
+
+| Class | Role |
+|-------|------|
+| `FASTPythonImporter` | Parse Python source string or file → FAST model. Extends `TSFASTAbstractImporter`. |
+| `FASTPythonTreeSitterVisitor` | Walks TreeSitter CST → builds FAST model. 1100+ lines, the core import logic. |
+| `FASTPythonCFGVisitor` | Builds control flow graph. Uses `FASTTCFGUtility` trait from FAST. |
+| `FASTPythonLocalResolverVisitor` | Links entity usages to their local declarations (scope-aware). |
+| `FASTPythonSSAVisitor` | SSA transform. Uses `FASTCFGTVisitor` trait from FAST. Requires local resolution first. |
+| `FASTPythonMetamodelGenerator` | Generates the Famix metamodel. Run via `FASTPythonMetamodelGenerator new generate`. |
+
+**Typical analysis pipeline:**
+```smalltalk
+model := FASTPythonImporter parseFile: aFile.
+FASTPythonLocalResolverVisitor resolve: model module.
+model allFunctionDefinitions first cfg.          "CFG"
+FASTPythonSSAVisitor resolve: model allFunctionDefinitions first.  "SSA (after resolution)"
+```
+
+### Variable-analysis API
+
+- Python-specific variable APIs (`usedVariables`, `isResolvedVariable`, `transitiveAssignedExpressions`, `transitiveAssignedExpressionsMap`, `internalAccesses`, `allNodesUsingMe`, `statementsUsingMe`, `allNodesUsingMyVersion`, `statementsUsingMyVersion`, `callsOnVariable`, `callsOnVariableVersion`) are single implementations on `FASTPyEntity` in `src/FAST-Python-Tools/FASTPyEntity.extension.st` (protocol `*FAST-Python-Tools`). The transitive ones require SSA resolution; the others require local resolution. The `*MyVersion` and `*VariableVersion` variants also require SSA.
+- Contrast: FAST-level helpers (`versionWriteAccesses`, `assignedExpressionsMap`, `versionAccesses`, ...) follow a mass-extension convention -- one identical copy per FAST class in protocol `*FAST-Core-Tools`, living in the FAST dependency repo, not here.
+
+## Testing
+
+- **Base test class**: `FASTPythonAbstractTestCase` (in Tools-Tests, tag `Core`) -- provides a `parse:` helper that wraps `FASTPythonImporter` with error reporting. The `model` instance variable has no accessor.
+- **Variable-analysis tests**: `FASTPythonVariablesAnalysisTest` is a slim base providing `parseAndResolve:` (parse + local resolution + SSA). Four subclasses in tag `Analysis`: `FASTPythonAssignedExpressionsInVariablesTest`, `FASTPythonIsResolvedVariablesTest`, `FASTPythonUsedVariablesTest`, `FASTPythonInternalAccessesTest`. Since each class tests a single concept, test methods use the plain `tests` protocol -- no specialized protocols.
+- **Documentation**: `resources/doc/analysis.md` documents the local resolver, SSA, shadowing and variable-query APIs. Convention: questions are written as bold questions without new TOC entries, and only implemented API is documented.
+- **Importer tests**: `FASTPythonImporterTest` extends `TSFASTAbstractImporterTest` (from TreeSitter). The test file is ~53k lines -- generated tests for every AST node type. Test generation snippet is in the class comment.
+- **CFG/Resolver/SSA tests**: `FASTPythonCFGTest`, `FASTPythonLocalResolverTest`, `FASTPythonSSATest`.
+
+## Gotchas
+
+- **TreeSitter python grammar version matters**: After v0.25, expression statements were removed from the tree. If tests fail unexpectedly, check your `tree-sitter-python` version first (see README).
+- **Pharo 13 is the primary target**; Pharo 14 is also tested. The baseline references `FAST:v3` which requires Moose 13.
+- **The importer uses the master branch** of `TSLibrariesPython` (hardcoded in `FASTPythonTreeSitterVisitor class >> initialize`).
+- **`expression --|> statement`**: In Python, expressions can be expression statements. This is reflected in the model hierarchy and affects how the visitor processes nodes.
+- **CFG uses `FASTTCFGUtility`** from FAST (the parent project). SSA uses `FASTCFGTVisitor`. These traits define the core graph traversal protocol.
+- **Local resolver scope management**: The resolver maintains a `scopes` stack. Shadowing is handled by checking entity kind consistency (same kind = shared declaration, different kind = new declaration).
+- **Python 3 only for resolver**: The local resolver is Python 3 scoped. Python 2 comprehension scoping (no scope) is not supported (issue #26).
+- **Non-local declarations are not deduplicated**: Each access to an unknown field creates a separate `FASTNonLocalDeclaration` (issue #26 in the repo).
+- **In-image method installation**: the MCP `pharo_method_compile` can be blocked by critiques (e.g. `ReDeadBlockRule`). Fallback: `Behavior>>compile:classified:` with method source **without** the outer `[ ]` brackets -- with brackets the body parses as an inner block and a silent no-op wrapper method gets installed. Use CR line returns (see global conventions).
+- **The importer cannot parse CR line endings**: pass the string through `withPlatformLineEndings` before parsing to avoid errors.
+- **`assertCollection:hasSameElements:` is multiplicity-insensitive** in current images (`#(a a)` vs `#(a)` passes) -- add explicit `size` assertions to catch duplicates.
+- **`model` ivar of `FASTPythonAbstractTestCase` has no accessor** -- in DoIt/evaluate code use `instVarNamed: #model`.
diff --git a/src/BaselineOfFASTPython/BaselineOfFASTPython.class.st b/src/BaselineOfFASTPython/BaselineOfFASTPython.class.st
@@ -12,9 +12,9 @@ Class {
 BaselineOfFASTPython >> baseline: spec [
 
 	<baseline>
-	spec for: #common do: [ 
-			spec preLoadDoIt: #'preload:package:'.
-			
+	spec for: #common do: [
+			spec preLoadDoIt: #preload:package:.
+
 			"Dependencies"
 			self
 				fast: spec;
@@ -25,11 +25,13 @@ BaselineOfFASTPython >> baseline: spec [
 				package: 'FAST-Python-Model' with: [ spec requires: #( 'FAST' 'TreeSitter' ) ];
 				package: 'FAST-Python-Model-Generator';
 				package: 'FAST-Python-Tools' with: [ spec requires: #( 'FAST-Python-Model' ) ];
+				package: 'FAST-Python-ExtraTools' with: [ spec requires: #( 'FAST-Python-Tools' ) ];
 				package: 'FAST-Python-Tools-Tests' with: [ spec requires: #( 'FAST-Python-Tools' ) ].
 
 			"Groups"
 			spec
 				group: 'Core' with: #( 'FAST-Python-Model' 'FAST-Python-Tools' );
+				group: 'Profile' with: #( 'FAST-Python-ExtraTools' );
 				group: 'Generator' with: #( 'FAST-Python-Model-Generator' );
 				group: 'Tests' with: #( 'FAST-Python-Tools-Tests' ) ]
 ]

diff --git a/src/FAST-Python-ExtraTools/FASTPyEntity.extension.st b/src/FAST-Python-ExtraTools/FASTPyEntity.extension.st
@@ -0,0 +1,15 @@
+Extension { #name : 'FASTPyEntity' }
+
+{ #category : '*FAST-Python-ExtraTools' }
+FASTPyEntity >> uniqueIdentifier [
+
+	<FMProperty: #uniqueIdentifier type: #String>
+	<FMComment: 'An ID that will not change compared to the moose ID. This is used for the Profile project.'>
+	^ self attributeAt: #uniqueIdentifier ifAbsentPut: [ UUID new asString ]
+]
+
+{ #category : '*FAST-Python-ExtraTools' }
+FASTPyEntity >> uniqueIdentifier: aString [
+
+	^ self attributeAt: #uniqueIdentifier put: aString
+]
diff --git a/src/FAST-Python-ExtraTools/FASTPyModel.extension.st b/src/FAST-Python-ExtraTools/FASTPyModel.extension.st
@@ -0,0 +1,21 @@
+Extension { #name : 'FASTPyModel' }
+
+{ #category : '*FAST-Python-ExtraTools' }
+FASTPyModel >> exportAsJSONWithDerived [
+
+	^ self exportAsJSONWithDerivedBlacklist: #(  )
+]
+
+{ #category : '*FAST-Python-ExtraTools' }
+FASTPyModel >> exportAsJSONWithDerivedBlacklist: aCollection [
+	"The blacklist should be a collection of properties to exclude such as `TEntityMetalevelDependency>>fanIn`."
+
+	^ String streamContents: [ :stream |
+			  FMModelExporterWithDerived new
+				  blacklist: aCollection;
+				  model: ((FMModel withMetamodel: self metamodel)
+						   addAll: self entities;
+						   yourself);
+				  printer: (FMJSONPrinter on: stream);
+				  run ]
+]
diff --git a/src/FAST-Python-ExtraTools/FMModelExporterWithDerived.class.st b/src/FAST-Python-ExtraTools/FMModelExporterWithDerived.class.st
@@ -0,0 +1,60 @@
+"
+I am a special exporter that exports also derived properties. This is useful when we want to provide a model to people not using Moose.
+
+I also allow to give a blacklist of properties to exclude. It should contain identifier of properties to exclude in this form: `TEntityMetaLevelDependency>>#fanIn` for example.
+
+If a property fails to be computed, it is not exported.
+
+Example 
+
+```smalltalk
+model exportAsJSONWithDerivedBlacklist: #('TEntityMetaLevelDependency>>#isDead' 'TEntityMetaLevelDependency>>#fanIn' 'TEntityMetaLevelDependency>>#fanOut' 'TEntityMetaLevelDependency>>#numberOfDeadChildren' 'TEntityMetaLevelDependency>>#numberOfExternalClients' 'TEntityMetaLevelDependency>>#numberOfExternalProviders' 'TEntityMetaLevelDependency>>#numberOfInternalProviders' 'TEntityMetaLevelDependency>>#numberOfInternalClients')
+```
+
+"
+Class {
+	#name : 'FMModelExporterWithDerived',
+	#superclass : 'FMModelExporter',
+	#instVars : [
+		'blacklist'
+	],
+	#category : 'FAST-Python-ExtraTools',
+	#package : 'FAST-Python-ExtraTools'
+}
+
+{ #category : 'accessing' }
+FMModelExporterWithDerived >> blacklist [
+	^ blacklist
+]
+
+{ #category : 'accessing' }
+FMModelExporterWithDerived >> blacklist: anObject [
+	blacklist := anObject
+]
+
+{ #category : 'initialization' }
+FMModelExporterWithDerived >> initialize [
+
+	super initialize.
+	blacklist := OrderedCollection new
+]
+
+{ #category : 'exporting' }
+FMModelExporterWithDerived >> shouldExportProperty: property for: anEntity [
+
+	(self blacklist includes: property compiledMethod name) ifTrue: [ ^ false ].
+
+	^ [ super shouldExportProperty: property for: anEntity ]
+		  on: Error
+		  do: [
+				  property traceCr.
+				  false ]
+]
+
+{ #category : 'exporting' }
+FMModelExporterWithDerived >> shouldIgnoreProperty: property [
+
+	(model metamodel includes: property) ifFalse: [ ^ true ].
+
+	^ false
+]
diff --git a/src/FAST-Python-ExtraTools/ManifestFASTPythonExtraTools.class.st b/src/FAST-Python-ExtraTools/ManifestFASTPythonExtraTools.class.st
@@ -0,0 +1,17 @@
+"
+I am a package containing extra tools. Originally for Profile contract.
+"
+Class {
+	#name : 'ManifestFASTPythonExtraTools',
+	#superclass : 'PackageManifest',
+	#category : 'FAST-Python-ExtraTools-Manifest',
+	#package : 'FAST-Python-ExtraTools',
+	#tag : 'Manifest'
+}
+
+{ #category : 'asserting' }
+ManifestFASTPythonExtraTools class >> shouldBeIncludedByDefaultInMetamodelsWith: aCollectionOfPackages [
+	"Includes in MM that are based on FAST"
+
+	^ aCollectionOfPackages anySatisfy: [ :package | package definedClasses anySatisfy: [ :class | class = FASTPyEntity ] ]
+]
diff --git a/src/FAST-Python-ExtraTools/package.st b/src/FAST-Python-ExtraTools/package.st
@@ -0,0 +1 @@
+Package { #name : 'FAST-Python-ExtraTools' }
diff --git a/src/FAST-Python-Model/FASTPyAssertStatement.class.st b/src/FAST-Python-Model/FASTPyAssertStatement.class.st
@@ -47,7 +47,6 @@ FASTPyAssertStatement >> expressions [
 	<ignoreForCoverage>
 	<generated>
 	<FMComment: 'The expressions asserted'>
-	<derived>
 	^ expressions
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyChevron.class.st b/src/FAST-Python-Model/FASTPyChevron.class.st
@@ -62,7 +62,6 @@ FASTPyChevron >> parentPrintStatement [
 	<ignoreForCoverage>
 	<generated>
 	<container>
-	<derived>
 	^ parentPrintStatement
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyClassDefinition.class.st b/src/FAST-Python-Model/FASTPyClassDefinition.class.st
@@ -85,7 +85,6 @@ FASTPyClassDefinition >> keywords [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ keywords
 ]
 
@@ -103,7 +102,6 @@ FASTPyClassDefinition >> superclasses [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ superclasses
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyClassPattern.class.st b/src/FAST-Python-Model/FASTPyClassPattern.class.st
@@ -55,7 +55,6 @@ FASTPyClassPattern >> elements [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ elements
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyCollectionInitializer.class.st b/src/FAST-Python-Model/FASTPyCollectionInitializer.class.st
@@ -57,7 +57,6 @@ FASTPyCollectionInitializer >> initializers [
 	<ignoreForCoverage>
 	<generated>
 	<FMComment: 'Each initializer defines one element.'>
-	<derived>
 	^ initializers
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyComparisonOperator.class.st b/src/FAST-Python-Model/FASTPyComparisonOperator.class.st
@@ -52,7 +52,6 @@ FASTPyComparisonOperator >> operands [
 	<ignoreForCoverage>
 	<generated>
 	<FMComment: 'List of the operands of the comparison operands. For example if we have `a == b != c`, it will be a, b and c.'>
-	<derived>
 	^ operands
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyComprehension.class.st b/src/FAST-Python-Model/FASTPyComprehension.class.st
@@ -91,7 +91,6 @@ FASTPyComprehension >> conditions [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ conditions
 ]
 
@@ -109,7 +108,6 @@ FASTPyComprehension >> forClauses [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ forClauses
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyConcatenatedString.class.st b/src/FAST-Python-Model/FASTPyConcatenatedString.class.st
@@ -44,7 +44,6 @@ FASTPyConcatenatedString >> strings [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ strings
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyElseClause.class.st b/src/FAST-Python-Model/FASTPyElseClause.class.st
@@ -54,7 +54,6 @@ FASTPyElseClause >> parentIfStatement [
 	<ignoreForCoverage>
 	<generated>
 	<container>
-	<derived>
 	^ parentIfStatement
 ]
 

diff --git a/src/FAST-Python-Model/FASTPyExecStatement.class.st b/src/FAST-Python-Model/FASTPyExecStatement.class.st
@@ -63,7 +63,6 @@ FASTPyExecStatement >> scopes [
 
 	<ignoreForCoverage>
 	<generated>
-	<derived>
 	^ scopes
 ]