Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,10 +123,9 @@ A Valve is made in the pipeline file: `add_valve(predicate, filter)`. It
joins a Predicate to a Filter. The Filter runs only when
`predicate.test(payload)` is true. There is no Valve file.

A step must be a class in a module. A lambda, or a class in the script that
you run, is refused. The reason is step identity: a step is identified by its
type, for example `examples.pricing.discount_filter.DiscountFilter`. State,
Hooks and the export use that identity.
A step is a class. State and Hooks report each step by the name of its
class, for example `DiscountFilter`. A lambda is refused, because it does not
have the method of a step.

```bash
python -m examples.pricing
Expand Down
103 changes: 47 additions & 56 deletions conformance/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,23 +94,23 @@ A Valve is added with one call, the condition first:
- When `predicate.test(payload)` is true, the Filter runs. When it is false,
the Payload passes through unchanged and the step is recorded in
`state.skipped`, not `state.executed`.
- The step is identified by its Filter (see "Step identity").
- The step has the label of its Filter (see "Step labels").
- `add_valve` is the one way to add a Valve. There is no Valve class for a
user to make, and `add_filter` does not accept one.

## Tap — reads the Payload (Plane B)

`observe(payload)`, no return value. A Tap reads the Payload and does not
write to it. **A Tap failure never halts the pipeline:** the error is
recorded in `state.errors` under the Tap's identity, the Tap still counts as
recorded in `state.errors` under the Tap's label, the Tap still counts as
executed, and hooks are not notified.

## Hook — sees the lifecycle

`before(step, payload)`, `after(step, payload)`, `on_error(step, error, payload)`.
A Hook reads the Payload and does not write to it.

- `step` is the **step's identity** (see "Step identity"). Pipeline-level
- `step` is the **step's label** (see "Step labels"). Pipeline-level
events pass absent (`None`).
- `error` is the error itself (never just its message).
- Order on success: `before(pipeline)`, then for each filter or Valve
Expand All @@ -121,8 +121,8 @@ A Hook reads the Payload and does not write to it.

## State — what happened in the last run

`executed` (identities, in order), `skipped` (identities), `errors`
(identity + error), `timings` (identity → seconds, when timing is on). A run
`executed` (labels, in order), `skipped` (labels), `errors`
(label + error), `timings` (label → seconds, when timing is on). A run
starts with a new State. **A failed run keeps the steps that completed
before the failure.**

Expand All @@ -140,66 +140,57 @@ before the failure.**
| run (async, opt-in) | `run_async(p)` |
| last run | `state` |
| timing and lineage | `observe(timing=True, lineage=False)` |
| each step's identity, in order | `identities` |
| where each step was added (`file:line`) | `step_sites` |
| each step's label, in order | `labels` |

- **Steps have no names.** State and Hooks report each step's identity.
- **Steps are not named by hand.** State and Hooks report each step's label.
- **`run` is synchronous** and never starts an event loop. If it meets an
async step it raises `AsyncStepInSyncRun` rather than silently paying for
a loop.
- A nested pipeline is one step: its identity appears once in the parent's
- A nested pipeline is one step: its label appears once in the parent's
State.
- **Timing and lineage are off by default.**

## Step identity: the filter, not a name

A step's identity is a pointer to the code that runs it, derived
automatically: `add_filter(f)`, not `add_filter(f, name)`. State, Hooks and
exported code all report this identity.

Why: in a codebase maintained by agents and reviewed by humans, an identity
taken from the code cannot drift from it. A hand-written name is an
unchecked label that can describe code which no longer matches it; a pointer
to the filter is always true, more specific, and needs no one to maintain
it.

### The identity

- **The filter's fully qualified symbol** (`app.filters.Validate`), plus an
**occurrence number only when the same identity appears more than once**
in one pipeline: the second is `...Validate#2`. Stable across unrelated
edits.
- **Extra detail, not identity**: the source location of the `add_*` call
(`file:line`), for agents and reviewers.
- **Valves**: the Filter's identity; the predicate's symbol is extra detail.
- **Pipelines used as a step are named types**: `class UserOnboarding(Pipeline)`,
identified by that type's symbol exactly as a filter is. A plain, unnamed
`Pipeline` can run at the top level but cannot be nested.
- Renames and moves change the identity. That is honest (it is different
code); cross-version comparisons follow renames from version control.
- A filter defined in the entry script (`__main__.X`) is refused like an
anonymous one: its identity changes with how the program is started, and
tests cannot import it.

### Anonymous filters are unsupported

Lambdas, closures, classes defined inside a function, inline functions: as
filters, taps, valve predicates or nested pipelines. They are not
independently testable and have no stable identity. Enforced in two layers:

1. **The runtime refuses them** at `add_*` time (`AnonymousStep`), with a
message naming the fix.
2. **The linter flags them** at authoring time:
`python -m codeupipe_lint.anonymous_steps <paths...>`. One line per
anonymous step, `path:line:col CUP-A00N slug message`; exit 1 when it
finds any. An intentional one (a test of the refusal) is suppressed with
a `cup: allow-anonymous` comment. Rules and limits:
[packages/codeupipe-lint/README.md](../packages/codeupipe-lint/README.md).
## Step labels

A step is reported by a label. The label is the name of the step's class:
`add_filter(ValidateFilter())` gives the label `ValidateFilter`. State and
Hooks report this label. A step is not given a name by hand:
`add_filter(f)`, not `add_filter(f, name)`.

- **A second use gets a number.** When the same label is in one pipeline
more than once, the second is `ValidateFilter#2`, the third
`ValidateFilter#3`.
- **A Valve** has the label of its Filter.
- **A nested pipeline** has the label of its class. Make it a named type
(`class UserOnboarding(Pipeline)`), so that the label says what it is. A
plain `Pipeline` that is nested has the label `Pipeline`.
- A label is for people and for records. It does not say where the class is
defined, and two classes with the same name in different modules have the
same label.

### What is refused when a step is added

A step must have the method of its role. `add_filter` needs `call(payload)`,
`add_tap` needs `observe(payload)`, `add_valve` needs a predicate with
`test(payload)` and a Filter with `call(payload)`, and `add_pipeline` needs a
Pipeline. A value without that method (a lambda, a plain function) is
refused with a `TypeError` that names the method.

### What the linter flags

A step that is a lambda, a function, a class defined inside a function, or a
plain nested `Pipeline` cannot be imported and tested alone, and its label
says little. The linter flags these when the code is written:
`python -m codeupipe_lint.anonymous_steps <paths...>`. One line per finding,
`path:line:col CUP-A00N slug message`; exit 1 when it finds any. An
intentional one is suppressed with a `cup: allow-anonymous` comment. Rules
and limits:
[packages/codeupipe-lint/README.md](../packages/codeupipe-lint/README.md).

**Conformance.** The shared cases name their steps; the runner builds every
step, so it maps each reported identity back to the case's step name. An
identity that is wrong, or differs between the dynamic run and an export,
does not map and fails the comparison.
step, so it maps each reported label back to the case's step name. A label
that is wrong, or differs between the dynamic run and an export, does not
map and fails the comparison.

## Not in the contract yet

Expand Down
2 changes: 1 addition & 1 deletion conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ Numbers compare as JSON numbers (`1 == 1.0`), but `true` is never `1`.
| tap | `fail` message | raise |
| valve `when` | `equals` key, value | `get(key) == value` |
| valve `when` | `truthy` key | `get(key)` is truthy (missing ⇒ false) |
| hooks | `"hooks": true` | record `before:<step>`, `after:<step>`, `error:<step>`; pipeline-level events use an empty step name. On a filter failure: `error:<step>` then `error:`. The runner maps each reported step identity (API.md) back to the case's step name |
| hooks | `"hooks": true` | record `before:<step>`, `after:<step>`, `error:<step>`; pipeline-level events use an empty step name. On a filter failure: `error:<step>` then `error:`. The runner maps each reported step label (API.md) back to the case's step name |

## Runner protocol

Expand Down
2 changes: 1 addition & 1 deletion conformance/runners/python/cup_ops/nested_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@

class NestedPipeline(Pipeline):
"""The runner's named pipeline type: nested pipelines must be named
types (conformance/API.md, step identity)."""
types (conformance/API.md, step labels)."""
2 changes: 1 addition & 1 deletion conformance/runners/python/cup_ops/record_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

class RecordHook:
"""Records ``before:<step>``, ``after:<step>``, ``error:<step>``; the
pipeline-level events use an empty step. Hooks receive step identities;
pipeline-level events use an empty step. Hooks receive step labels;
the runner maps them back to the case's step names (runner.labels)."""

def before(self, step, payload):
Expand Down
8 changes: 4 additions & 4 deletions conformance/runners/python/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,15 +130,15 @@ def build(case, cls, predicates, names):


def labels(p, names):
"""identity -> the case's step name, for the steps of ``p``.
"""label -> the case's step name, for the steps of ``p``.

Steps are identified by their filter (conformance/API.md), while the
Steps are reported by label (conformance/API.md), while the
shared cases name them. The runner built every step, so it knows which
case step each identity belongs to. An identity reported that is not in
case step each label belongs to. A label reported that is not in
this map (wrong, or inconsistent between modes) stays as is and fails
the comparison.
"""
return {identity: names[id(step)] for identity, step, _type in p._steps}
return {label: names[id(step)] for label, step, _type in p._steps}


def _named(steps, label):
Expand Down
12 changes: 6 additions & 6 deletions docs/static-export.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ Exit code: **0** exported, **1** refused (the message names the step and the
fix), **2** the command could not run. Nothing is written on 1 or 2. Tests:
[tests/lower/test_cli.py](../tests/lower/test_cli.py).

## Step identity in exported code
## Step labels in exported code

Steps are identified by their filter (`app.filters.Validate`, `#2` from the
second use; conformance/API.md). An identity is a label, not an identifier,
so exported code never uses it as a name: each step gets a generated module
and function, `step_01_validate`, `step_02_validate`, and the identity
appears only as a string in State and in Hook calls. That also rules out
A step has a label, the name of its class (`Validate`, `Validate#2` from the
second use; conformance/API.md). A label is not an identifier, so exported
code never uses it as a name: each step gets a generated module and
function, `step_01_validate`, `step_02_validate`, and the label appears only
as a string in State and in Hook calls. That also rules out
every collision between a step and a builtin, a module global or the
generated `run()` (scorecard findings F3–F8).

Expand Down
6 changes: 3 additions & 3 deletions examples/pricing/print_steps_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@


class PrintStepsHook(Hook):
"""Sees the lifecycle. ``step`` is the step's identity; it is None for
the pipeline itself."""
"""Sees the lifecycle. ``step`` is the label of the step, which is the
name of its class. It is None for the pipeline itself."""

def before(self, step, payload):
if step is not None:
print(" " + step.replace("examples.pricing.", ""))
print(" " + step)

def after(self, step, payload):
pass
Expand Down
23 changes: 9 additions & 14 deletions packages/codeupipe-lint/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,10 @@ in the list: each linter uses the ids that are its own.

## Anonymous-step linter

Step identity (conformance/API.md, "Step identity") makes a step's identity a
pointer to its filter. Anonymous steps have no such pointer and are
unsupported. The runtime refuses them when the step is added; this linter
flags them at authoring time.
A step is reported by the name of its class (conformance/API.md, "Step
labels"). A step that is a lambda, a function, a class defined inside a
function, or a plain nested `Pipeline` has no useful name and cannot be
imported and tested alone. This linter flags them when the code is written.

```bash
python -m codeupipe_lint.anonymous_steps <paths...> # files or directories
Expand All @@ -40,7 +40,7 @@ python -m codeupipe_lint.anonymous_steps --rules # list the rule i
Each finding is one line, and the message names the fix:

```
path:line:col CUP-A001 lambda-step Valve predicate is a lambda. Define it at module level (...) so it has a stable identity and can be tested on its own.
path:line:col CUP-A001 lambda-step Valve predicate is a lambda. Define it at module level (...) so it has a name in State and can be tested on its own.
```

With no paths, both linters read them from `cup.config.json`
Expand All @@ -61,7 +61,6 @@ returns the result Payload (`findings`, `suppressed`, `errors`, `report`, `exit_
| CUP-A002 | local-function-step | a named function declared inside a function |
| CUP-A003 | local-type-step | a class declared inside a function, or an instance of one |
| CUP-A005 | unnamed-pipeline-step | a plain, unnamed `Pipeline` nested as a step; nested pipelines must be named Pipeline types |
| CUP-A006 | entry-script-step | a class or function of a script (`__main__`), added from its `if __name__ == "__main__":` block |
| CUP-A008 | function-predicate | a named function as a Valve predicate. Predicates are objects with `test(payload)` (conformance/API.md, "Valve") |

A step is any argument of `add_filter`, `add_tap` or `add_pipeline` (on any
Expand All @@ -75,19 +74,15 @@ A comment containing `# cup: allow-anonymous` (any text may follow)
suppresses a finding when it is on the finding's line, on the line of the
enclosing add or Valve call, or alone on the line directly above. Suppressed
findings are counted in the summary. Use it only where the anonymous step is
the point, as in the tests that exercise the runtime's refusal
(`tests/core/test_step_identity.py`).
the point, as in a test of what the runtime does with one
(`tests/core/test_step_labels.py`).

### Limits

Names resolve lexically (innermost function scope out to the module; a
parameter, loop target or import shadows). A name bound two different ways
in one scope is not flagged. CUP-A006 is deliberately conservative: only a
module with a `__main__` guard, only steps added from that block or from a
module-level function it calls (transitively); a script without a guard is
not flagged (the runtime still refuses it). Not seen: `functools.partial`,
`type(...)`-built classes, steps built in another module (all refused at
runtime).
in one scope is not flagged. Not seen: `functools.partial`,
`type(...)`-built classes, steps built in another module.

### Layout

Expand Down
Loading
Loading