Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,14 @@ safety argument stays structural:
down, need one added to a bound, so they are accepted only where that cannot
overflow (`plusOne`): a constant, or something minus a positive constant. A
constant limit of zero or less, or a negative constant start, would put a negative
constant index in the emitted code and is rejected. `Trip` states every loop as
the forward `Start <= i < Limit`; a limit that is not exactly `len` of the stored
slice makes every operand slice carry an explicit cap (`Plan.Bound`), or the last
vector would write past it.
constant index in the emitted code and is rejected. A constant written with
literals alone (`16`, `1 - 1`) is spelled as its value; one that names a constant
keeps the name (`spell`), since the emitted file is compiled for every build and
the name can stand for another value in another (a file for each GOARCH). Checks
that need a constant read `Trip.StartValue`/`LimitValue`, never the text. `Trip`
states every loop as the forward `Start <= i < Limit`; a limit that is not exactly
`len` of the stored slice makes every operand slice carry an explicit cap
(`Plan.Bound`), or the last vector would write past it.
- A slice operand is a variable or a chain of field selections ending in a slice
(`x.f`, `x.f.g`) whose root variable the body does not assign (`sliceRef`). The body has
no call and stores only elements, so nothing in it changes the header a field holds. The
Expand All @@ -107,9 +111,13 @@ safety argument stays structural:
iteration writes, so the scalar loop sees the old value and the emitter loads it
before any store of the chunk (`statements` hoists those loads). A negative
offset is a recurrence (`ReasonCarriedDependence`); a variable distance or a
loop that counts down is `ReasonOffsetOfStored`. The length check is
against the last index read (`lastIndex`), and offset loads are not capped at the
limit. Every other store and load is `ident[i]` (`index`) — `dst[i+1]`, `dst[2*i]` are rejected outright, regardless of
loop that counts down is `ReasonOffsetOfStored`. A constant offset that puts an
index below zero, the lowest one read with a constant start or every one with a
constant limit, is `ReasonNegativeIndex`: the scalar loop panics and the emitted
code would index at a negative constant. A positive one that pushes a constant
start or limit past the largest int is rejected as that bound, since the emitted
code adds them. The length check is against the last index read (`lastIndex`),
and offset loads are not capped at the limit. Every other store and load is `ident[i]` (`index`) — `dst[i+1]`, `dst[2*i]` are rejected outright, regardless of
whether they're actually dependent. That is what guarantees no two
iterations touch the same element.
- A non-slice operand becomes an `Invariant` only if it is a variable, a
Expand Down
6 changes: 6 additions & 0 deletions internal/loopir/ir.go
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,12 @@ type Trip struct {
// Start and Limit are loop-invariant integer expressions, as source text.
// Start is "0" for the common case.
Start, Limit string
// StartConst and LimitConst are set when Start and Limit are constants, whose
// values are StartValue and LimitValue. The text of a constant that names
// another constant keeps the name (see spell), so the checks that need a
// constant read these.
StartConst, LimitConst bool
StartValue, LimitValue int64
// Slice is the slice whose length is Limit, when Limit is exactly len(Slice).
Slice *Ref
// NonEmpty is set when both are constants and Start < Limit, so the loop runs
Expand Down
139 changes: 87 additions & 52 deletions internal/loopir/lower.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ func (l *lowerer) rangeLoop(s *ast.RangeStmt) (*Loop, Reason) {
l.valueVar = l.info.Defs[valueIdent]
l.valueRef = ref
}
return l.body(s, s.Body, Trip{Start: "0", Limit: "len(" + ref.Text() + ")", Slice: ref}, 1)
return l.body(s, s.Body, Trip{Start: "0", StartConst: true, Limit: "len(" + ref.Text() + ")", Slice: ref}, 1)
}
if s.Value != nil {
return nil, ReasonUnsupportedRange
Expand Down Expand Up @@ -215,7 +215,7 @@ func (l *lowerer) reverse(s *ast.ForStmt, start, limit ast.Expr, strict bool) (*
if !isConst || low == math.MaxInt64 {
return nil, ReasonUnsupportedBound
}
from = bound{text: strconv.FormatInt(low+1, 10), value: low + 1, isConst: true}
from = bound{text: spell(limit, low, 1), value: low + 1, isConst: true}
} else if from, ok = l.bound(limit); !ok {
return nil, ReasonUnsupportedBound
}
Expand Down Expand Up @@ -269,10 +269,14 @@ type bound struct {
// trip states a loop that runs from to to (exclusive) as a Trip.
func (l *lowerer) trip(from, to bound) Trip {
return Trip{
Start: from.text,
Limit: to.text,
Slice: to.slice,
NonEmpty: from.isConst && to.isConst && from.value < to.value,
Start: from.text,
Limit: to.text,
StartConst: from.isConst,
LimitConst: to.isConst,
StartValue: from.value,
LimitValue: to.value,
Slice: to.slice,
NonEmpty: from.isConst && to.isConst && from.value < to.value,
}
}

Expand All @@ -287,10 +291,7 @@ func (l *lowerer) bound(e ast.Expr) (bound, bool) {
b := bound{text: types.ExprString(e)}
b.value, b.isConst = l.constant(e)
if b.isConst {
// A named constant or a constant expression (s0, 1 - 1) is spelled as its
// value, so the checks that read a constant start or limit see it and the
// emitted code never indexes at a negative constant they would have caught.
b.text = strconv.FormatInt(b.value, 10)
b.text = spell(e, b.value, 0)
}
if call, ok := ast.Unparen(e).(*ast.CallExpr); ok {
b.slice, _ = l.lenOf(call)
Expand All @@ -302,29 +303,57 @@ func (l *lowerer) bound(e ast.Expr) (bound, bool) {
// need. It is only accepted where adding one cannot overflow: a constant below the
// largest int, or something minus a positive constant.
func (l *lowerer) plusOne(e ast.Expr) (bound, bool) {
if v, isConst := l.constant(e); isConst {
if v == math.MaxInt64 {
return bound{}, false
}
return bound{text: strconv.FormatInt(v+1, 10), value: v + 1, isConst: true}, true
}
sub, ok := ast.Unparen(e).(*ast.BinaryExpr)
if !ok || sub.Op != token.SUB {
return bound{}, false
}
k, isConst := l.constant(sub.Y)
if !isConst || k < 1 || !l.pureInt(sub.X) {
v, isConst := l.constant(e)
if isConst && v == math.MaxInt64 {
return bound{}, false
}
// x - k + 1 is x - (k-1); with k == 1 that is x itself.
x, ok := l.bound(sub.X)
if !ok {
return bound{}, false
// x - k + 1 is x - (k-1); with k == 1 that is x itself. A constant that names
// another is written this way too, so width - 1 + 1 is width. One written with
// literals alone is spelled as its value.
if sub, ok := ast.Unparen(e).(*ast.BinaryExpr); ok && sub.Op == token.SUB && !literal(e) {
if k, kConst := l.constant(sub.Y); kConst && k >= 1 {
if x, ok := l.bound(sub.X); ok {
if k == 1 {
return x, true
}
return bound{text: x.text + " - " + strconv.FormatInt(k-1, 10), value: v + 1, isConst: isConst}, true
}
}
}
if k == 1 {
return x, true
if isConst {
return bound{text: spell(e, v, 1), value: v + 1, isConst: true}, true
}
return bound{text: x.text + " - " + strconv.FormatInt(k-1, 10)}, true
return bound{}, false
}

// spell is the source text of the constant e, whose value is value, plus delta. A
// constant written with literals alone (16, 1 - 1) is spelled as its value. One
// that names a constant keeps the name, since the emitted file is compiled for
// every build and the name can stand for another value in another one (a file for
// each GOARCH).
func spell(e ast.Expr, value, delta int64) string {
if literal(e) {
return strconv.FormatInt(value+delta, 10)
}
// Every integer operator binds at least as tightly as +, and operators that bind
// as tightly group from the left, so the sum needs no parentheses.
text := types.ExprString(e)
if delta != 0 {
text += " + " + strconv.FormatInt(delta, 10)
}
return text
}

// literal reports whether e is written with literals and operators alone, no names.
func literal(e ast.Expr) bool {
found := false
ast.Inspect(e, func(n ast.Node) bool {
if _, ok := n.(*ast.Ident); ok {
found = true
}
return !found
})
return !found
}

// constant returns the value of e when it is an integer constant that fits an int64.
Expand Down Expand Up @@ -516,6 +545,17 @@ func (l *lowerer) body(node ast.Stmt, block *ast.BlockStmt, trip Trip, step int6
// the chunk. A read of an earlier iteration's element (d < 0) is a recurrence,
// and a distance that is not a constant, or a loop that counts down, has no
// order loopvec can check.
//
// A constant offset that puts an index below zero makes the scalar loop panic
// when it runs. Start is the lowest index the loop visits, whichever way it
// counts, so a constant start puts the lowest index read at Start+off, and a
// constant limit puts the highest at Limit-1+off. Such a loop is left alone,
// since the vector loop would index or slice at that negative constant, which
// does not compile. Start is never negative and Limit never below one (see
// rangeReason), so only a negative offset can do this, and it is at least
// -maxOffset, so negating it cannot overflow. A positive offset that pushes a
// constant start or limit past the largest int would not compile either, as the
// emitted code adds them.
stored := map[types.Object]bool{}
for _, stmt := range body {
if store, ok := stmt.(Store); ok {
Expand All @@ -525,35 +565,30 @@ func (l *lowerer) body(node ast.Stmt, block *ast.BlockStmt, trip Trip, step int6
for _, stmt := range body {
for _, leaf := range Leaves(stmtValue(stmt)) {
load, ok := leaf.(*Load)
if !ok || load.Off == "" || !stored[load.Ref.Obj] {
if !ok || load.Off == "" {
continue
}
distance, err := strconv.ParseInt(load.Off, 10, 64)
switch {
case err != nil || step != 1:
return nil, ReasonOffsetOfStored
case distance < 0:
return nil, ReasonCarriedDependence
if stored[load.Ref.Obj] {
switch {
case err != nil || step != 1:
return nil, ReasonOffsetOfStored
case distance < 0:
return nil, ReasonCarriedDependence
}
}
}
}

// A constant offset that puts an index below zero makes the scalar loop panic
// when it runs: on its first iteration when the start is a constant and the
// first index is negative, or on every iteration when the limit is a constant
// and even the last index is. It is left alone, since the vector loop would
// index or slice at a negative constant, which does not compile.
start, errStart := strconv.ParseInt(trip.Start, 10, 64)
limit, errLimit := strconv.ParseInt(trip.Limit, 10, 64)
negative := func(offset string) bool {
off, err := strconv.ParseInt(offset, 10, 64)
return err == nil && (errStart == nil && start+off < 0 || errLimit == nil && limit+off <= 0)
}
for _, stmt := range body {
for _, leaf := range Leaves(stmtValue(stmt)) {
if load, ok := leaf.(*Load); ok && negative(load.Off) {
if err == nil && distance < 0 &&
(trip.StartConst && trip.StartValue < -distance || trip.LimitConst && trip.LimitValue <= -distance) {
return nil, ReasonNegativeIndex
}
if err == nil && distance > 0 {
switch {
case trip.StartConst && trip.StartValue > math.MaxInt64-distance:
return nil, ReasonUnsupportedStart
case trip.LimitConst && trip.LimitValue > math.MaxInt64-distance:
return nil, ReasonUnsupportedBound
}
}
}
}

Expand Down
35 changes: 9 additions & 26 deletions internal/loopir/plan.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
package loopir

import (
"go/types"
"strconv"
)
import "go/types"

// maxSlices is the most distinct slices a loop may touch. Each stored slice is
// checked at run time against every other slice in the loop, so the number of
Expand Down Expand Up @@ -166,22 +163,18 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) {
p.Stmts = append(p.Stmts, planned)
}
if len(p.Stmts) == 1 && p.Stmts[0].Dst != nil {
if load, ok := p.Stmts[0].Root.(*Load); ok {
p.Copy = !negativeSource(l.Ind.Trip.Start, load.Off)
}
_, p.Copy = p.Stmts[0].Root.(*Load)
}

trip := l.Ind.Trip
// A loop that runs a few times, known at compile time, is faster as it is.
if from, errFrom := strconv.ParseInt(trip.Start, 10, 64); errFrom == nil {
if to, errTo := strconv.ParseInt(trip.Limit, 10, 64); errTo == nil {
least := int64(minTrip)
if p.Full {
least = int64(MinReduceVectors * MaxLanes(elem))
}
if to-from < least {
return nil, ReasonShortLoop
}
if trip.StartConst && trip.LimitConst {
least := int64(minTrip)
if p.Full {
least = int64(MinReduceVectors * MaxLanes(elem))
}
if trip.LimitValue-trip.StartValue < least {
return nil, ReasonShortLoop
}
}
p.Start, p.Limit, p.NonEmpty, p.TripRef = trip.Start, trip.Limit, trip.NonEmpty, trip.Slice
Expand Down Expand Up @@ -228,16 +221,6 @@ func NewPlan(l *Loop, opts Options) (*Plan, Reason) {
return p, ""
}

// negativeSource reports whether a copy would slice its source at a negative
// constant, which does not compile: the loop starts at start and reads at offset
// off, and both are constants that add up to less than zero. Such a loop panics
// when it runs, which the ordinary vector loop does as well.
func negativeSource(start, off string) bool {
s, errStart := strconv.ParseInt(start, 10, 64)
o, errOff := strconv.ParseInt(off, 10, 64)
return errStart == nil && errOff == nil && s+o < 0
}

// checkOps returns why simd cannot express an operation in v on the element
// type, or "".
func checkOps(elem types.Type, v Value) Reason {
Expand Down
17 changes: 11 additions & 6 deletions internal/loopir/reason.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@ const (
ReasonUnsupportedRange Reason = "range clause is not i := range s, i, v := range s, or i := range n"
// ReasonUnsupportedStart covers a counting loop whose start is not a
// loop-invariant integer expression (a constant, an int variable, len of a
// slice, and sums and products of those), or is a negative constant.
// slice, and sums and products of those), is a negative constant, or is a
// constant that a read at a positive constant offset pushes past the largest
// int.
ReasonUnsupportedStart Reason = "loop start is not a loop-invariant integer expression"
// ReasonUnsupportedCondition covers a loop condition that is not
// i < limit or i <= limit, or i >= limit or i > limit when counting down.
Expand All @@ -28,7 +30,9 @@ const (
// ReasonUnsupportedBound covers a limit that is not a loop-invariant integer
// expression, is a constant that is not positive, or is written so that
// adding one to it could overflow (a <= limit, or the start of a loop that
// counts down, that is not a constant or something minus a constant).
// counts down, that is not a constant or something minus a constant), or is
// a constant that a read at a positive constant offset pushes past the
// largest int.
ReasonUnsupportedBound Reason = "loop limit is not a loop-invariant integer expression"
// ReasonUnsupportedDestination covers a destination that isn't a plain
// slice[i] element: an offset (dst[i+1]), a stride (dst[2*i]), or a
Expand All @@ -43,10 +47,11 @@ const (
// wrote, so the iterations cannot run as a vector.
ReasonCarriedDependence Reason = "an iteration reads an element an earlier iteration wrote"
// ReasonNegativeIndex covers a slice read at a constant offset that puts an
// index below zero: the first index, with a constant start (a[i-1] in a loop
// from 0), or every index, with a constant limit (a[i-20] in a loop that stops
// before 16). The scalar loop panics when it runs, and the vector loop would
// index or slice at a negative constant, which does not compile.
// index below zero: the one read at the lowest index the loop visits, with a
// constant start (a[i-1] in a loop over 0..n-1, whichever way it counts), or
// every one, with a constant limit (a[i-20] in a loop that stops before 16). The
// scalar loop panics when it reaches that index, and the vector loop would index
// or slice at a negative constant, which does not compile.
ReasonNegativeIndex Reason = "a constant offset puts an index below zero"
// ReasonUnsupportedIf covers an if that is not a select: a condition that is
// not a comparison of loop values (joined by && and ||), an init statement, or
Expand Down
Loading
Loading