[halimi]

Summary

References: https://github.com/TheThingsIndustries/lorawan-stack-support/issues/1517

Downlink scheduling on LoRa Basics Station gateways could fail with sub band is blocked for up to hours after the concentrator time jumped backwards. This happens on gateways that do not extend the concentrator time beyond 32 bits (the time restarts from zero every 2^32 µs, ~71.6 minutes), and on gateways whose concentrator time base changes when they reconnect. The emissions recorded in the scheduler stayed in the old time base, so they appeared to be in the future and blocked their sub-band.

Changes

• When the scheduler synchronizes with the gateway concentrator time and the new concentrator time lags more than one minute behind the time expected from the server time, it shifts all recorded emissions (scheduler-wide and per sub-band) to the new time base. The emissions keep counting towards the duty-cycle at their actual position, but no longer block the sub-band. A warning is logged when this happens.
• Add a WithIgnorePriorUplinkTokens connection option, which makes the scheduler ignore uplink tokens of uplink messages received before the connection was established. The LoRa Basics Station frontend uses it, so that a downlink scheduled right after a reconnect cannot synchronize the new connection's clock to the previous connection's time base. Other frontends are unchanged, because their concentrator time does not depend on the connection.

Testing

Steps

• go test ./pkg/gatewayserver/scheduling/... ./pkg/gatewayserver/io/...
• TestScheduleConcentratorTimeJump covers a 32-bit rollover that the gateway did not account for, a time base reset, and normal jitter that must not trigger a shift.
• TestScheduleSyncViaUplinkToken and TestIgnorePriorUplinkTokens cover ignoring uplink tokens from before the connection was established.

Results

All tests in pkg/gatewayserver/scheduling and pkg/gatewayserver/io/... pass. With the emission shift disabled, TestScheduleConcentratorTimeJump fails with sub band is blocked for 1h11m34.9s (rollover) and sub band is blocked for 94h0m1.95s (time base reset).

Regressions

• Duty-cycle enforcement: emissions are shifted, not dropped, so they still count towards the duty-cycle after a jump.
• Clock synchronization for frontends other than LoRa Basics Station is unchanged. The Semtech UDP frontend keeps using RolloverClock.Sync, which already treats a backwards timestamp as a rollover.
• Small backwards jitter (LoRa Basics Station xtime can go back up to ~100 ms) stays below the one-minute tolerance and does not trigger a shift.

Notes for Reviewers

The shift happens during clock synchronization, before any scheduling, so the MaxScheduleAhead checks see the emissions in the new time base.

Checklist

• Scope: The referenced issue is addressed, there are no unrelated changes.
• Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md for the chosen target branch.
• Documentation: Relevant documentation is added or updated.
• Testing: The steps/process to test this feature are clearly explained including testing for regressions.
• Infrastructure: If infrastructural changes (e.g., new RPC, configuration) are needed, a separate issue is created in the infrastructural repositories.
• Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
• Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

A critical race in scheduler garbage collection allows synchronization to shift emissions after the cutoff is calculated, potentially deleting recent transmissions and future reservations and losing duty-cycle and collision accounting.

1 open issue

• Hold mutex across GC cutoff calculation and pruning · New

What changed in this PR

This PR addresses prolonged downlink blocking when a LoRa Basics Station gateway’s concentrator clock jumps backwards, while preserving duty-cycle accounting.

Changes:

• Shift recorded emissions into the new time base when the clock falls more than one minute behind expectations.
• Ignore pre-connection uplink tokens when synchronizing LoRa Basics Station gateway clocks.
• Add regression tests and document the fixes in the changelog.

File	Description
pkg/​gatewayserver/​scheduling/​sub_band.go	Add synchronized shifting of sub-band emissions.
pkg/​gatewayserver/​scheduling/​scheduler.go	Detect backward clock jumps, shift emissions, and filter stale uplink tokens.
pkg/​gatewayserver/​scheduling/​scheduler_test.go	Cover clock resets, rollover, jitter, and stale tokens.
pkg/​gatewayserver/​scheduling/​emission.go	Add an operation to shift emission timestamps.
pkg/​gatewayserver/​io/​semtechws/​ws.go	Enable stale-token filtering for LoRa Basics Station connections.
pkg/​gatewayserver/​io/​io.go	Forward the connection option to the scheduler.
pkg/​gatewayserver/​io/​io_test.go	Test connection-level stale-token filtering.
CHANGELOG.md	Document clock-jump recovery and stale-token handling.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the changes addressing downlink blocking after backwards clock jumps.

No open issues 🎉

1 resolved since last review

• Hold mutex across GC cutoff calculation and pruning

_{Review effort: Balanced}

[johanstokking]

Thanks. I reviewed the code and it looks fine. I don't understand the cause.

This happens on gateways that do not extend the concentrator time beyond 32 bits (the time restarts from zero every 2^32 µs, ~71.6 minutes)

They all rollover right? In case of Basic Station, we have two fields in xtime: the session ID and the 32-bit concentrator timestamp:

lorawan-stack/pkg/gatewayserver/io/semtechws/time.go

Lines 109 to 119 in 71f91c5

	// TimestampFromXTime constructs the concentrator timestamp associated with the
	// provided XTime.
	func TimestampFromXTime(xTime int64) uint32 {
	// The concentrator timestamp is the 32 LSB.
	return uint32(xTime & 0xFFFFFFFF)
	}
	// SessionIDFromXTime constructs the session ID associated with the provided XTime.
	func SessionIDFromXTime(xTime int64) int32 {
	return int32(xTime >> 48)
	}

Since you cannot reproduce the issue, I think what we need to collect is better data: we need full logging on the LBS LNS, including xtime and downlink scheduling.

[This happens] on gateways whose concentrator time base changes when they reconnect

When they reconnect, we create a new scheduler and start all over with empty emissions right?

[halimi]

Thanks. I reviewed the code and it looks fine. I don't understand the cause.

This happens on gateways that do not extend the concentrator time beyond 32 bits (the time restarts from zero every 2^32 µs, ~71.6 minutes)

They all rollover right? In case of Basic Station, we have two fields in xtime: the session ID and the 32-bit concentrator timestamp:

lorawan-stack/pkg/gatewayserver/io/semtechws/time.go

Lines 109 to 119 in 71f91c5

	// TimestampFromXTime constructs the concentrator timestamp associated with the
	// provided XTime.
	func TimestampFromXTime(xTime int64) uint32 {
	// The concentrator timestamp is the 32 LSB.
	return uint32(xTime & 0xFFFFFFFF)
	}
	// SessionIDFromXTime constructs the session ID associated with the provided XTime.
	func SessionIDFromXTime(xTime int64) int32 {
	return int32(xTime >> 48)
	}

Since you cannot reproduce the issue, I think what we need to collect is better data: we need full logging on the LBS LNS, including xtime and downlink scheduling.

Agreed, we need more data to better understand the issue. Especially if it only happens on one kind of gateway.

[This happens] on gateways whose concentrator time base changes when they reconnect

When they reconnect, we create a new scheduler and start all over with empty emissions right?

Correct, a reconnect creates a fresh scheduler. But what could happen is that a fresh scheduler has no clock sync, so a downlink right after the reconnect whose uplink token comes from the previous connection syncs the new clock to the old time base, and the emission is recorded there. The first uplink then moves the clock back, and if the difference was big between the two timestamps then that emission blocks the sub-band.

[johanstokking]

The first uplink then moves the clock back, and if the difference was big between the two timestamps then that emission blocks the sub-band.

Yes, that is a good point and that is a good fix here. This is basically relevant for all frontends, not just LBS LNS.

Maybe that is the only culprit here. And it's an understandable one, i.e. when using Class C downlink with an uplink token that was from before a restart + reconnect.

Can you carve that out in a new PR, or reduce the scope of this PR? And also add debug logging so we can collect the right information.

[halimi]

Can you carve that out in a new PR, or reduce the scope of this PR? And also add debug logging so we can collect the right information.

Yes, I opened a new PR for it #8080