Skip to content

Configured waiting queue capacity is smaller than LOCALJEV_MAX_QUEUE #2

Description

@dajiaohuang

On main at 3f23e36, Engine.decide compares total in-flight plus waiting decisions with LOCALJEV_MAX_QUEUE, while README documents that setting as waiting decisions before HTTP 529. With maxInflight=1 and maxQueue=1, a deterministic mock holds the first inference request and the second call is immediately rejected as overloaded, leaving the configured queue slot unused. Expected: allow up to maxQueue waiting decisions in addition to up to maxInflight active decisions. Actual: maxQueue limits total admitted decisions, shrinking usable queue space by the active count. This causes earlier overload responses under load; no interface or default needs to change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions