feat(flow): count look-alike rows on native and remote surfaces

.github/workflows/packages.yml

@@ -549,6 +549,75 @@ jobs:
           sys.exit(1 if bad else 0)
           PY
 
+  # Real native list apps read through the row census (identical rows halt,
+  # record ids pick the row, a unique row runs): a WPF ListBox over UI
+  # Automation on Windows and a Swift NSTableView over AX on macOS. Both legs
+  # must run every real-app test; the step summary lists each outcome.
+  flow-native-census:
+    name: flow native row census (${{ matrix.os }})
+    needs: scope
+    if: needs.scope.outputs.flow == 'true'
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 25
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [windows-latest, macos-latest]
+    defaults:
+      run:
+        working-directory: packages/flow
+        shell: bash
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+          cache: pip
+      - name: Install
+        run: |
+          pip install -e ../types -e . pytest pytest-timeout
+          if [ "$RUNNER_OS" = "Windows" ]; then pip install uiautomation; fi
+          if [ "$RUNNER_OS" = "macOS" ]; then
+            pip install pyobjc-framework-ApplicationServices pyobjc-framework-Cocoa pyobjc-framework-Quartz
+          fi
+      - name: Census unit tests and real-app tests
+        env:
+          OPENADAPT_TEST_WINDOWS_UIA: "1"
+          OPENADAPT_TEST_MACOS_AX: "1"
+        run: |
+          mkdir -p runs
+          pytest -v -rs tests/test_native_identity_census.py \
+            tests/test_native_census_real_apps.py --junitxml=runs/native-junit.xml
+      - name: The real-app tests ran (no skips)
+        run: |
+          python - <<'PY'
+          import os
+          import sys
+          import xml.etree.ElementTree as ET
+          prefix = {"Windows": "test_windows_wpf_", "macOS": "test_macos_table_"}[
+              os.environ["RUNNER_OS"]
+          ]
+          cases = [
+              c for c in ET.parse("runs/native-junit.xml").getroot().iter("testcase")
+              if c.get("name", "").startswith(prefix)
+          ]
+          lines = [f"### Native row census, real app ({os.environ['RUNNER_OS']})", ""]
+          for case in cases:
+              skip = case.find("skipped")
+              failed = case.find("failure") is not None or case.find("error") is not None
+              state = "skipped: " + (skip.get("message") or "") if skip is not None else (
+                  "failed" if failed else "passed"
+              )
+              lines.append(f"- `{case.get('name')}`: {state}")
+          with open(os.environ["GITHUB_STEP_SUMMARY"], "a", encoding="utf-8") as out:
+              out.write("\n".join(lines) + "\n")
+          print("\n".join(lines))
+          skipped = [c.get("name") for c in cases if c.find("skipped") is not None]
+          sys.exit(1 if not cases or skipped else 0)
+          PY
+
   capture-linux:
     name: capture (linux, py${{ matrix.python-version }})
     needs: scope
@@ -1033,6 +1102,7 @@ jobs:
       - flow-lint
       - flow-unit
       - flow-critical
+      - flow-native-census
       - capture-linux
       - capture-macos
       - capture-windows

llms-full.txt

@@ -1342,6 +1342,38 @@ For an armed step:
   run's value into the recorded band and requires the whole band to match. A
   band that rests on a glyph-confusable identifier keeps the pixel comparison.
 
+A verified identity says the row reads like the recorded row, not that no
+other row reads the same. Before an armed step acts, the runtime counts the
+rows that show the same identity (`openadapt_flow/runtime/identity_census.py`). When more
+than one does, it acts only on the row with the record id recorded at teach
+time, or halts. How each surface counts:
+
+- browser: every shown row in the page, with app-owned `data-*` ids and row
+  links as record ids;
+- Windows (UI Automation agent): the sibling rows of the clicked row, with
+  non-positional AutomationIds as record ids. A grid that reports more rows
+  than it exposes, or a scrollable list that builds items on demand, cannot be
+  counted, and a row pick before a write halts;
+- macOS (AX): the rows of the `AXTable`, `AXOutline` or `AXList` around the
+  element, with the AX identifier and URL as record ids. A table that reports
+  more `AXRows` than it exposes, or a list with no row count inside a scroll
+  area with a scroll bar, cannot be counted;
+- Linux (AT-SPI): the sibling list items, table rows or tree items of the
+  clicked row, with AT-SPI accessible ids as record ids. A list with no Table
+  row count inside a scroll pane with a scroll bar cannot be counted. GTK 3
+  list boxes and
+  tree views expose no row identity under the structured reading (one child
+  per row, or cells without a row), so the identity check there rests on OCR;
+- pixel-only displays: the OCR rows of the frame, or, with no OCR band, every
+  place the identifier crop matches. Only a clear pixel mismatch rules a place
+  out; a frame at another scale than the recording cannot be counted.
+
+Ids that name a position (`row-3`, `r3c2`, `item_12`, the row's own index)
+never tell rows apart.
+
+A surface that cannot count the rows halts any row pick that leads into a
+write.
+
 On a pixel-only display, the OCR tier also stops the correct record whenever
 its own identifier contains a look-alike glyph (`O`/`0`, `l`/`1`, `B`/`8`).
 A deployment can remove most of that cost by giving the runtime a complete

packages/flow/docs/LIMITS.md

@@ -185,6 +185,38 @@ For an armed step:
   run's value into the recorded band and requires the whole band to match. A
   band that rests on a glyph-confusable identifier keeps the pixel comparison.
 
+A verified identity says the row reads like the recorded row, not that no
+other row reads the same. Before an armed step acts, the runtime counts the
+rows that show the same identity (`openadapt_flow/runtime/identity_census.py`). When more
+than one does, it acts only on the row with the record id recorded at teach
+time, or halts. How each surface counts:
+
+- browser: every shown row in the page, with app-owned `data-*` ids and row
+  links as record ids;
+- Windows (UI Automation agent): the sibling rows of the clicked row, with
+  non-positional AutomationIds as record ids. A grid that reports more rows
+  than it exposes, or a scrollable list that builds items on demand, cannot be
+  counted, and a row pick before a write halts;
+- macOS (AX): the rows of the `AXTable`, `AXOutline` or `AXList` around the
+  element, with the AX identifier and URL as record ids. A table that reports
+  more `AXRows` than it exposes, or a list with no row count inside a scroll
+  area with a scroll bar, cannot be counted;
+- Linux (AT-SPI): the sibling list items, table rows or tree items of the
+  clicked row, with AT-SPI accessible ids as record ids. A list with no Table
+  row count inside a scroll pane with a scroll bar cannot be counted. GTK 3
+  list boxes and
+  tree views expose no row identity under the structured reading (one child
+  per row, or cells without a row), so the identity check there rests on OCR;
+- pixel-only displays: the OCR rows of the frame, or, with no OCR band, every
+  place the identifier crop matches. Only a clear pixel mismatch rules a place
+  out; a frame at another scale than the recording cannot be counted.
+
+Ids that name a position (`row-3`, `r3c2`, `item_12`, the row's own index)
+never tell rows apart.
+
+A surface that cannot count the rows halts any row pick that leads into a
+write.
+
 On a pixel-only display, the OCR tier also stops the correct record whenever
 its own identifier contains a look-alike glyph (`O`/`0`, `l`/`1`, `B`/`8`).
 A deployment can remove most of that cost by giving the runtime a complete

packages/flow/openadapt_flow/backends/linux_backend.py

@@ -585,6 +585,25 @@ def locate_structural(
             supported_operations=list(candidate.supported_operations),
         )
 
+    def identity_candidates_at(self, x: int, y: int) -> Optional[dict[str, Any]]:
+        """Census of the rows that read the same as the row at (x, y).
+
+        See :class:`openadapt_flow.backend.IdentityCandidatesBackend`. Uses the
+        client's optional ``row_census``; a client without it cannot count
+        rows (None). Offsets are global pixels, the same scale as clicks.
+        """
+        census_at = getattr(self._client, "row_census", None)
+        if not callable(census_at):
+            return None
+        try:
+            window, global_x, global_y = self._global_point(
+                int(x), int(y), require_active=False
+            )
+            census = census_at(window, global_x, global_y, limit=2000)
+        except Exception:
+            return None
+        return census if isinstance(census, dict) else None
+
     def displayed_text_at(self, x: int, y: int) -> Optional[str]:
         """The text of the one AT-SPI element under (x, y), for a read-back.
 
@@ -1358,12 +1377,7 @@ def structured_text(self, element: LinuxElement) -> Optional[str]:
         own = element.native
         node = own
         row = None
-        row_roles = {
-            "list item",
-            "table cell",
-            "table row",
-            "tree item",
-        }
+        row_roles = self._ROW_ROLES
         for _ in range(8):
             if self._raw_role(node) in row_roles:
                 row = node
@@ -1384,6 +1398,13 @@ def structured_text(self, element: LinuxElement) -> Optional[str]:
                 break
             branch = parent
 
+        return self._row_text(row, branch)
+
+    _ROW_ROLES = frozenset({"list item", "table cell", "table row", "tree item"})
+
+    def _row_text(self, row: Any, branch: Any) -> Optional[str]:
+        """A row's identity: its children's text except the target's branch."""
+
         parts: list[str] = []
         for child in self._children(row):
             if child is branch:
@@ -1403,6 +1424,45 @@ def structured_text(self, element: LinuxElement) -> Optional[str]:
                 parts.append(text)
         return _clean_text(" ".join(parts))
 
+    def row_census(
+        self, window: LinuxWindow, x: int, y: int, *, limit: int
+    ) -> Optional[dict[str, Any]]:
+        """Census of the rows that read the same as the row at (x, y).
+
+        Mirrors :meth:`structured_text`: the record row is the nearest list
+        item, table row or cell, tree item; its identity is its children's
+        text except the target's branch; every sibling row is read the same
+        way. Record ids are AT-SPI accessible ids on the row and the target's
+        branch. A table whose Table interface reports more rows than it
+        exposes, or a row whose children are not built, is ``incomplete``.
+        Offsets are in global screen pixels. None on any failure.
+        """
+        try:
+            from openadapt_flow.runtime.identity_census import tree_row_census
+
+            element = self.element_at_point(window, x, y)
+            if element is None:
+                return None
+            client = self
+
+            def identity(row: Any, path: list[int]) -> Optional[str]:
+                if not path:
+                    return None
+                children = client._children(row)
+                if path[0] >= len(children):
+                    return None
+                return client._row_text(row, children[path[0]])
+
+            return tree_row_census(
+                element.native,
+                _AtspiTreeView(self),
+                row_roles=self._ROW_ROLES,
+                identity=identity,
+                max_rows=limit,
+            )
+        except Exception:
+            return None
+
     def perform_native(self, element: LinuxElement, operation: str) -> str:
         if operation == "focus":
             component = self._call(
@@ -1526,3 +1586,54 @@ def physical_scroll(self, dx: int, dy: int) -> bool:
         # not provide a reliable pixel-delta contract. Refuse rather than
         # fabricate delivery evidence.
         return False
+
+
+class _AtspiTreeView:
+    """``identity_census.TreeView`` over live AT-SPI nodes (screen pixels)."""
+
+    def __init__(self, client: AtspiLinuxClient) -> None:
+        self._client = client
+
+    def parent(self, node: Any) -> Any:
+        return self._client._call(node, "get_parent", default=None)
+
+    def children(self, node: Any) -> list[Any]:
+        return self._client._children(node)
+
+    def role(self, node: Any) -> Optional[str]:
+        return self._client._raw_role(node) or None
+
+    def origin(self, node: Any) -> Optional[tuple[float, float]]:
+        bounds = self._client._bounds(node)
+        return (float(bounds[0]), float(bounds[1])) if bounds is not None else None
+
+    def id_attributes(self, node: Any) -> list[tuple[str, str]]:
+        value = self._client._accessible_id(node)
+        return [("atspi-id", value)] if value else []
+
+    def row_count(self, container: Any) -> Optional[int]:
+        table = self._client._call(
+            container, "get_table_iface", "queryTable", default=None
+        )
+        if table is None:
+            return None
+        count = self._client._call(table, "get_n_rows", default=None)
+        if count is None:
+            count = getattr(table, "nRows", None)
+        try:
+            return int(count) if count is not None else None
+        except (TypeError, ValueError):
+            return None
+
+    def scrollable(self, container: Any) -> Optional[bool]:
+        """True when an enclosing scroll pane holds a scroll bar."""
+        node = container
+        for _ in range(3):
+            if node is None:
+                return False
+            if self.role(node) == "scroll pane":
+                return any(
+                    self.role(child) == "scroll bar" for child in self.children(node)
+                )
+            node = self.parent(node)
+        return False