Merge workspace, amplitude threshold splits, and curation workflow improvements

[rossant]

Summary

Add reversible merge workspaces, AIND/SpikeInterface merge-proposition review, and amplitude-threshold splitting for the Template GUI. Preserve complete selection, table, and history context through curation operations, and return successful merges to Cluster View with only the result selected for immediate quality assignment.

Merge and proposition workflows

• V enters or cancels Merge mode; G merges all staged clusters and selected Similarity candidates, including selections hidden by filtering. Cancellation restores the entry workspace; undo restores the exact pre-merge workspace, and redo restores the result.
• Ordinary right-click transfers the clicked table row or correlogram row cluster between the primary and Similarity roles in Normal, Merge, and proposition workflows. Drag-and-drop transfers and reorders staged candidates. Transferring a reference promotes the next primary cluster and retains the old reference in Similarity; the last primary reference is protected. Modified right-click is not a transfer alias.
• Merge View keeps one persistent dock identity, layout, and geometry. Scientific views follow staged order and then selected Similarity rows in visible table order. Successful merges close Merge mode and reveal the result without changing Cluster View sort order.
• Read format-version 2 merges from producer-owned curation.json. The persistent proposition queue uses stable source-order labels, concise rows, status colors, and full-detail tooltips. Clicking a pending proposition stages it for review; candidates can be added, removed, or reordered before acceptance.
• Alt+Down / Alt+Up navigate pending propositions; Alt+Backspace rejects and advances; Alt+Shift+Backspace resets a completed review. G commits and returns to Cluster View; navigation to the next proposition is explicit after acceptance.
• Accept/reject/reset and merge undo/redo preserve exact review context. Edited acceptance records accepted_modified; overlapping proposals become stale when source units disappear. Save decisions and applied/result IDs atomically to curation_review.json, coupled to the normal dirty/save lifecycle and protected by a source hash. Never overwrite curation.json.

Splitting, selection, and GUI behavior

• Alt-right-drag in Amplitude View previews a threshold for one selected cluster, linked to individual waveform traces. K commits an exact all-spike partition strictly below the threshold; empty and whole-cluster selections are rejected. Control-right-click or the view menu clears the preview.
• Keep a stable blue reference, selection colors, and display order. Support repeated Control+Space Similarity batches and a persistent Skip Noise and MUA preference. Restore selection and table context through merge, split, metadata changes, and history.
• Enter a bare integer in an enabled table filter to select and reveal that exact ID. Persist independently rearranged Cluster, Similarity, and Merge columns by name.
• Correct firing-rate units/binning; share seconds/minutes/hours recording-time preferences. Show dirty state, enable Save only for pending curation, and confirm successful saves.
• Add a searchable in-GUI shortcut reference, clearer view menus, stable native table sizing, narrower merge docks, and responsive correlogram labels/hit-testing.
• Support per-cluster/shared spike budgets, defer hidden-tab rendering, and maintain per-cluster indexes to limit interactive work on large datasets. Release owned Qt/event callbacks on shutdown.
• Correct template-less stored waveform/channel handling so Waveform and Probe views use cluster-specific channels. Preserve the waveform-type rescaling fix and cover W / Shift+W with actual Qt key events.

Dependencies and release acceptance

Require released phylib>=2.7.1,<3, regenerate the lockfile from PyPI, and remove the temporary test-CI archive substitution. Keep the intentional scheduled upstream compatibility job. Update installation instructions and historical handoff notes.

Cyrille approved the handoff and merges on 8 October 2026, accepting documented automated real-dataset validation as maintainer acceptance in place of the historical named-tester gate. This PR develops phy 2.2.0.dev0; it does not publish a phy release.

Validation

• Final local validation (8 October 2026): make test-full passed 495 core/GUI tests and 155 application tests, with one existing visual benchmark skipped; aggregate coverage 97%. Lint, formatting, make doc-check and uv build passed. Every tracked file in this integrated branch matches the fully tested combination with phylib 2.7.1, Tornado 6.5.10 and fsspec 2026.6.0.
• Fresh installation of the built phy wheel automatically resolves phylib 2.7.1 from PyPI. Package imports, version checks, dependency checks, and template-describe pass without editable checkouts or a phylib override.
• Automated GUI smoke tests on isolated copies of the real SWAY dataset verify exact merge assignments, undo/redo, return to Normal mode, immediate quality assignment, save/reopen with templates removed, and preservation of producer-owned curation.json. Raw recording paths are unavailable, so these exercise stored waveforms rather than raw trace extraction; they are not human visual reviews.
• Final head c3fd5cc0b4ef8e23cbadee80cf234bb1711bd8ee: CI and spelling passed before merge. Squash merged as cae2a1e9b5bff0eca89ae1f05248161d4f3168ef; the merged tree is identical to the tested head. Resulting master CI passed the complete Linux/macOS/Windows Python 3.10–3.12 matrix, docs and packaging; master spelling also passed.
• Separate service check: GitHub-managed Dependency Graph submission failed before job execution, with no logs or annotations. GitHub rejects rerunning that dynamic managed workflow; its cause remains unconfirmed.

Try the source build

Use a separate checkout and a copy of a dataset:

git clone https://github.com/cortex-lab/phy.git phy-merge-workflow
cd phy-merge-workflow
git switch master
uv sync --frozen --dev
uv run phy template-gui /path/to/copied/dataset/params.py

The stable PyPI phy release remains 2.1.0 until a separate phy release is prepared.

[goatsofnaxos]

In version current as of Aug 19 the keyboard shortcuts to switch between different waveforms types (single, average, templates) no longer works. The functionality is available by clicking in the WaveformView window but it no longer works as a keyboard shortcut.

[rossant]

Strange. Any error message in the terminal? There's no text box that is stealing the keyboard focus? Are other shortcuts working?

[goatsofnaxos]

Strange. Any error message in the terminal? There's no text box that is stealing the keyboard focus? Are other shortcuts working?

No errors in the terminal. OTher shortcuts (e.g. space bar) are working.

[rossant]

Integration update:

• Merged current origin/master in 0eed29a7; PR conflicts are resolved.
• Preserved both waveform-type rescaling and transient spike-highlight cleanup.
• Added a real Qt key-event regression for Waveform View W / Shift+W; both shortcuts work on the current branch. The August shortcut report is not reproducible at current HEAD.
• Local validation: make lint, make format-check, and make doc-check passed; 486 core/GUI tests passed with one documented visual-benchmark skip.
• Released phylib 2.7.0 still reproduces the known template-less waveform failure. With the CI-pinned phylib 2.7.1.dev0 revision, all 153 application tests pass.
• Fixed stale uv Git-cache handling for the temporary phylib pin. Fresh CI is green across Linux, macOS, and Windows on Python 3.10-3.12; docs, codespell, and wheel build also pass.

The PR remains a draft pending the phylib 2.7.1 release, replacement of the temporary Git pin with phylib>=2.7.1, regenerated lock data, and final real-dataset acceptance.