Skip to content

docs(integration-toolkit): large data loads, the 503 for exhausted capacity, and sandbox organizations - #190

Merged
karlrank merged 4 commits into
mainfrom
docs/bulk-loads-and-fair-processing
Nov 20, 2026
Merged

karlrank merged 4 commits into
mainfrom
docs/bulk-loads-and-fair-processing

Conversation

@karlrank

@karlrank karlrank commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Replaces #182.

  • New page Large Data Loads (/integrations/integration-toolkit/large-data-loads): how migrations, full resyncs and large file imports are recognized and processed, what an integrator notices, and what a client has to handle:
    • 503 with Retry-After: 60 and BULK_CAPACITY_EXHAUSTED when the capacity for large loads is in use. Nothing of the request is queued; in a mixed request the interactive events are accepted and come back skipped on the retry.
    • 503 with Retry-After: 2 and INGEST_THROTTLED when the allowance cannot be checked or the receiving queue throttles. Nothing was accepted, the same request is sent again, no duplicates.
    • 422 with EVENT_TOO_WIDE above 5,000 updates per event and REQUEST_TOO_WIDE above 25,000 updates per request. The page says plainly that staging and production only observe these limits today.
    • The order of events across the start and the end of a large load, and the one exception (the pause reaching its time limit).
    • Events that produce very many updates, managed calls (never refused) and file imports (wait every 60 seconds for up to 6 hours, progress.waiting_for_capacity).
  • New page Sandbox Organizations (/integrations/integration-toolkit/sandbox-organizations), rewritten from docs(integration-toolkit): sandbox organizations and the SANDBOX_LANE monitoring code #182 for the new behaviour: everyday traffic is handled as in production, large loads in a sandbox share a small capacity and run at a low rate. The Integration Hub banner and the SANDBOX_LANE notice link to this address.
  • inbound/getting-started.md: a "When to Retry" table for 429 and the two 503 responses, and a pointer to the size limits.
  • interactive-inbound.md: 503 added where retries are described; interactive events do not count towards the allowance and are never refused for lack of capacity.
  • monitoring/investigating.md: a replay can be answered with 503, or with 200 and Retry-After when it was throttled part of the way.
  • overview.md, configuration.md: cross-links.

monitoring/codes.md is generated and is not changed here. The new monitoring code EVENT_TOO_WIDE arrives there with the next npm run update-monitoring-codes.

Status of the feature. It is implemented and merged in the Integration Toolkit, tested on dev, and switched off on every stage. It is not in production. Both pages carry a "Not active yet" note and describe what happens once it is on, without a date.

Do not merge before the feature is live, or decide first that the "Not active yet" notes are enough to publish ahead of it. The notes have to be removed when it is switched on, and the note on the size limits when they are enforced in production.

Checked: npm run build passes, and the anchors the pages link to exist in the built page.

Update 2026-11-20: bulk import windows, and large loads are live

  • New section Bulk Import Windows (Integration Hub card in the Monitoring tab, and GET/PUT/DELETE /v1/bulk-imports/window). Planning a Migration now recommends booking one.
  • Large loads have been processed this way in production since 2026-11-19: the "Not active yet" notes are removed, and the size limits are enforced.
  • When a load ends: 15 minutes without events, or 3 minutes for a trickle after a load of at least 30 minutes.

Merge only after the API (integration-toolkit-api !333) and the Hub card (epilot360-integration-hub !91) are in production.

Karl Rankla added 4 commits November 13, 2026 15:47
…pacity, and sandbox organizations

Adds a Large Data Loads page: how everyday traffic and large loads are
processed, what a large load looks like from outside, the 503 response with
Retry-After and BULK_CAPACITY_EXHAUSTED (whole and mixed requests), file
imports that wait for capacity, and planning a migration.

Adds the Sandbox Organizations page at the address the Integration Hub banner
and the SANDBOX_LANE notice link to, for the new sandbox behaviour.

Adds the 503 where retries are described (inbound getting started, interactive
inbound closing rules and rate limits, replay) and cross-links from the
overview and the configuration page.
… import

Quotes the reworded 503 messages: the one for a request refused whole, the
top-level message of a mixed request and the message on each refused event,
and the new message of an import that gave up waiting.

Shows a file import that waits for capacity through
progress.waiting_for_capacity (since, retry_at, gives_up_at), with an example
job, and drops the sentence that its progress simply does not advance.
…rged behaviour

- 503 INGEST_THROTTLED (Retry-After: 2) beside 503 BULK_CAPACITY_EXHAUSTED (Retry-After: 60)
- size limits: 422 EVENT_TOO_WIDE above 5,000 updates per event, 422 REQUEST_TOO_WIDE above 25,000 per request, observed only in staging and production for now
- order of events across the start and the end of a large load, and the time limit that is the exception
- events that produce very many updates, managed calls, file imports waiting every 60 seconds for up to 6 hours
- replays: 503 when nothing was queued, 200 with Retry-After when throttled part of the way
- the processing is switched off in every environment: the pages say so
- New section Bulk Import Windows: booking a window in the Integration Hub
  (Monitoring tab) or with GET/PUT/DELETE /v1/bulk-imports/window, the rules,
  the lane status, the permissions. Planning a Migration now says to book one.
- Large loads have been processed this way in production since 2026-11-19:
  the "Not active yet" notes are gone, and so is the note that production
  only observes the size limits (they are enforced).
- When a load ends: after 15 minutes without events, or after 3 minutes when
  only a trickle follows a load that ran for at least 30 minutes.
- Sandboxes: a window works there too, at the sandbox rate.
@karlrank
karlrank marked this pull request as ready for review November 20, 2026 11:11
@karlrank
karlrank merged commit fb3d39f into main Nov 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant