Skip to content

Device login: concurrent polls of an approved code hold every database connection, and the instance stops answering #794

Closed

Description

@schmitch

Problem

poll (crates/server/src/auth/device.rs:561-611) opens a transaction and locks the request row FOR UPDATE on its connection (device_auth_dao::lock_for_poll, crates/db/src/device_auth_dao.rs:131-150). An approved request goes on to collect (device.rs:609), still holding the lock and the connection, and collect asks the pool for a second one:

  • user_dao::find_user_by_id(&state.pool, …) (device.rs:620);
  • for a scope that names repositories, repository_dao::find_repository_by_name(&state.pool, …) (device.rs:240-247, reached from :661).

Every other poll of the same device_code takes a connection in forge_db::begin and then blocks in PostgreSQL on the row lock, with no lock_timeout. An approved request is never answered slow_down, because too_fast is checked only while it is pending (device.rs:583-608), and the token endpoint has no rate limit.

Scenario

  1. Any signed-in user starts a device login and approves it on /device (device_login is on by default).
  2. A script sends 25 concurrent POST /auth/device/token requests with that device_code. The default pool is 20 connections with a 10 s acquire timeout.
  3. One poll holds the lock and waits for a second connection. The other 19 hold the rest of the pool while they wait for the lock.
  4. After 10 s the lock holder fails with 503. The next poll takes the lock and needs a second connection again.
  5. This repeats until the request expires (10 minutes), and the user can approve more codes. Meanwhile every database-backed request on the instance, feeds and sign-ins included, waits 10 s and fails.

Holding one pooled connection while waiting for another can also deadlock the pool without any malice, given enough concurrent collects.

Expected

  • collect reads the user and resolves the scope on the transaction's own connection, or before taking the lock.
  • lock_for_poll uses FOR UPDATE NOWAIT (or SKIP LOCKED), so a concurrent poll answers slow_down at once instead of holding a connection while it waits.

Test in crates/server/tests/device_auth.rs: with database.max_connections = 2, poll one approved code six times at once. Exactly one poll gets the token, every poll answers within a few seconds, and an unrelated request is served meanwhile.

New with #649 (#717). Found in the 2.2.0 release review.

Activity

  1. added this to the 2.3.0 milestone on Nov 2, 2026
  2. modified the milestones: 2.3.0, 2.2.0 on Nov 3, 2026
  3. self-assigned this
    on Nov 3, 2026
  4. added a commit that references this issue on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions