diff --git a/docs/integrations/integration-toolkit/configuration.md b/docs/integrations/integration-toolkit/configuration.md index 8ea79beb..4601e07e 100644 --- a/docs/integrations/integration-toolkit/configuration.md +++ b/docs/integrations/integration-toolkit/configuration.md @@ -693,6 +693,8 @@ alerting are documented in the [Monitoring section](./monitoring/overview.md). | Event submission | 100 events per request | | API requests | 1000 requests per minute per organization | +Large loads, such as migrations and full resyncs, are processed on capacity set aside for them. See [Large Data Loads](./large-data-loads.md), including the responses a client has to handle and the limits on the size of one event and one request. This processing is not active yet; the page says what is in effect today. + ## Security ### Authentication diff --git a/docs/integrations/integration-toolkit/inbound/getting-started.md b/docs/integrations/integration-toolkit/inbound/getting-started.md index 9cc747c1..04038483 100644 --- a/docs/integrations/integration-toolkit/inbound/getting-started.md +++ b/docs/integrations/integration-toolkit/inbound/getting-started.md @@ -122,6 +122,22 @@ the response carries a generated id instead. `status` of the event (`success`, `skipped` for a duplicate, `ignored` or `error`, the last three with a `message`) and its `event_id`. +### When to Retry + +Three responses ask you to send a request again: + +| Status | Meaning | What to do | +|---|---|---| +| `429` | Some events could not be accepted right now: their deduplication check was throttled, or another request with the same `deduplication_id` is still being processed. Those events have `status: "error"`. | Wait for the `Retry-After` interval, then send the whole request again. Events with a `deduplication_id` that this request accepted come back `skipped`. | +| `503` with `BULK_CAPACITY_EXHAUSTED` | Your organization went over its allowance while the capacity for large loads was in use. Nothing of the request was queued, apart from interactive events. | Wait for the `Retry-After` interval (60 seconds), then send the same request again. See [Large Data Loads](../large-data-loads.md#when-capacity-is-exhausted). | +| `503` with `INGEST_THROTTLED` | Ingestion was busy for a moment: your organization's allowance could not be checked, or the queue that receives the events throttled the request. If the body has no `results`, nothing of the request was accepted. | Wait for the `Retry-After` interval (2 seconds), then send the same request again. See [When Ingestion Is Busy](../large-data-loads.md#when-ingestion-is-busy). | + +Give every event a `deduplication_id`, so that a retried event is never processed twice. + +A `422` with the code `EVENT_TOO_WIDE` on an event, or `REQUEST_TOO_WIDE` on the request, is not a reason to retry: the event or the request produces too many updates and has to be split. See [Size Limits](../large-data-loads.md#size-limits). + +The two `503` responses and the size limits belong to the processing of [large data loads](../large-data-loads.md), which is not active yet. That page says what is in effect today. + ## Processing Flow Events flow through the following pipeline: @@ -139,4 +155,5 @@ portal's correlation id and `interactive: true` with the request, and let the po - [Unique Identifiers](./unique-identifiers.md) - Configure entity lookup strategies - [Relations](./relations.md) - Link entities together - [Interactive Inbound](../interactive-inbound.md) - Push data an end customer is waiting for +- [Large Data Loads](../large-data-loads.md) - How migrations and full resyncs are processed - [Examples](./examples.md) - See complete integration examples diff --git a/docs/integrations/integration-toolkit/interactive-inbound.md b/docs/integrations/integration-toolkit/interactive-inbound.md index b5551e01..eabb57aa 100644 --- a/docs/integrations/integration-toolkit/interactive-inbound.md +++ b/docs/integrations/integration-toolkit/interactive-inbound.md @@ -15,7 +15,7 @@ Your push stays asynchronous. Each push carries the **correlation id** the porta - An end customer is waiting for your push to land: registration, contract addition, or customer identification in a journey. - Your middleware answers a [portal extension hook](/docs/apps/components/portal-extension) such as `registrationIdentifiersCheck` or `contractIdentification` and then pushes the customer's data. -Bulk and scheduled syncs do not need it and keep using the inbound API as they are. +Bulk and scheduled syncs do not need it and keep using the inbound API as they are. See [Large Data Loads](./large-data-loads.md) for how large ones are processed. ::: ## How It Works @@ -202,7 +202,7 @@ curl -X POST 'https://integration-toolkit.sls.epilot.io/v3/erp/updates/events' \ }' ``` -"Accepted" means the request answered with success for every event. If a push is answered with a retry (a 422 with partial results, or a 429), retry it until it is accepted before you send the close. Give every event a `deduplication_id`, so a retried request — a closing one included — is recognized as the same request and not processed a second time. +"Accepted" means the request answered with success for every event. If a push is answered with a retry (a 422 with partial results, a 429, or a [503](./large-data-loads.md#what-to-do)), retry it until it is accepted before you send the close. Give every event a `deduplication_id`, so a retried request — a closing one included — is recognized as the same request and not processed a second time. If you cannot place the close correctly, leave it out: a close that arrives too early does more harm than none. The response says whether the close was applied; see [Closing Rules](#closing-rules). @@ -247,7 +247,7 @@ An empty `events` array is accepted only together with `correlation_complete: tr | Field | Description | |---|---| | `correlation_id` | The request's correlation id, echoed, or generated when you sent none — so a push without a portal in the loop still gets an id it can look up later. | -| `correlation_close` | Present when the request carried `correlation_complete: true`, on a 200, 422 or 429 alike. Says whether the close was applied; see [What Became of the Close](#what-became-of-the-close). | +| `correlation_close` | Present when the request carried `correlation_complete: true`, on a 200, 422 or 429 alike, and on a 503 that lists `results`. Says whether the close was applied; see [What Became of the Close](#what-became-of-the-close). | | `results[].interactive` | `true` when the event was **processed as interactive**: it was recorded and the wait can see it. Present when interactive processing was requested, or the use case is set to `always`. | | `results[].lane` | The lane that actually carried the event: `interactive`, or the organization's normal lane (`priority`, `default` or `deferred`). Over the [rate limit](#rate-limits) an event is still `"interactive": true` but runs on a normal lane. | | `results[].message` | Why `interactive` is `false`. | @@ -267,7 +267,7 @@ A request that does not ask for interactive processing is processed exactly as a - **The close needs a request-level `correlation_id`.** It closes that correlation and no other; the correlation is never inferred from per-event ids. `correlation_complete: true` without a request-level `correlation_id` is rejected with **400** and raises `CORRELATION_ID_MISSING`. - **The request's own events count.** The correlation closes after the request's events are recorded. Closing twice is a no-op. -- **Only a fully accepted request closes.** A response that asks you to retry (a 422 with partial results, or a 429), or a request one of whose events the queue refused, closes nothing — so a close never overtakes events that are still being retried. +- **Only a fully accepted request closes.** A response that asks you to retry (a 422 with partial results, a 429, or a 503), or a request one of whose events the queue refused, closes nothing — so a close never overtakes events that are still being retried. - **A retried close still closes.** If the response to your closing request was lost and you send it again with the same `deduplication_id`s, the events are skipped as duplicates and the close is applied to the correlation they were recorded under the first time. - **A request with events never creates a correlation.** It closes one that exists, whatever became of its own events — also when some went over the 32-event limit. If the correlation does not exist yet and none of the request's events was recorded under it (the use case does not allow interactive processing, `interactive` was left out, or every event carried its own `correlation_id`), the close is ignored: closing an empty correlation would tell the portal "nothing is coming" while the data is being written. - **A request without events creates the correlation closed.** For a correlation id epilot has never seen, it returns 200, raises `CORRELATION_CLOSED_EMPTY`, and the portal's wait answers `complete` with `events_received: 0` at once. Creating the correlation costs one unit of the [interactive budget](#rate-limits); closing one that exists costs nothing. @@ -299,7 +299,7 @@ Every request that sets `correlation_complete: true` gets a `correlation_close` | Reason | What to do | |---|---| -| `retry_requested` | The response is a 422 or a 429. Retry the request with `correlation_complete` still set; it closes then. | +| `retry_requested` | The response is a 422, a 429 or a 503. Retry the request with `correlation_complete` still set; it closes then. | | `queue_refused` | One of the request's events was refused. Send the close again once every event is accepted. | | `other_integration` | The correlation belongs to another integration. Use a correlation id of your own. | | `events_under_own_correlation_ids`, `events_duplicate`, `events_not_configured`, `events_not_interactive`, `events_unrecorded` | The correlation does not exist, and this request recorded none of its events under it: they carried their own correlation ids, were duplicates, matched no use case, or were not processed as interactive. Send the close under the correlation the events were recorded under, or send the events with `interactive: true` to a use case that allows it. | @@ -679,6 +679,8 @@ Each item in `results` is the same correlation summary the wait returns, without The interactive lane stays fast because no single organization can fill it. Every organization has a budget for interactive events: by default a sustained **120 interactive events per 5 minutes, with a burst of 40**. Organizations with higher registration volume can agree a higher budget with epilot. +Events on the interactive lane do not count towards an organization's allowance for [large data loads](./large-data-loads.md#what-counts), and they are never refused for lack of capacity. + How a request is charged: - **Once per request, at ingest, for the events it admits as interactive.** Duplicates, events rejected with 422 and events over a correlation's 32-event limit cost nothing, so a retry pays only for the events it newly admits. @@ -686,11 +688,12 @@ How a request is charged: - **All or nothing.** If the budget cannot cover the whole charge, all of the request's events run on the normal lane. A request is never split across lanes. - **A close without events** costs one unit, and only when it creates the correlation. Closing a correlation that exists costs nothing. When the budget is spent, such a close still closes a correlation that exists, but creates none, and `correlation_close` says `rate_limited`. -When the budget is spent, nothing is rejected and nothing is lost: +When the budget is spent, the events are not rejected because of that, and nothing is lost: - The events are processed on the organization's normal lane. Their results keep `"interactive": true`, and `lane` names the lane that carried them. - They are still recorded, so the close, the wait and the status endpoints keep working. The answer simply takes as long as a bulk event does, and the portal's wait may time out and retry. - `INTERACTIVE_RATE_LIMITED` appears in monitoring for the request. +- They count as everyday traffic for [large data loads](./large-data-loads.md). In the rare case that the organization goes over its allowance while the capacity for large loads is exhausted, or that its allowance cannot be checked, they are refused with `503` like any other event. Send them again after the `Retry-After` interval. A steady stream of `INTERACTIVE_RATE_LIMITED` usually means that a bulk sync sets `interactive: true` on every request. Bulk pushes are not slower without the flag, but with it they use up the budget your registrations need. diff --git a/docs/integrations/integration-toolkit/large-data-loads.md b/docs/integrations/integration-toolkit/large-data-loads.md new file mode 100644 index 00000000..d956e01a --- /dev/null +++ b/docs/integrations/integration-toolkit/large-data-loads.md @@ -0,0 +1,331 @@ +--- +sidebar_position: 8 +title: Large Data Loads +description: How the Integration Toolkit processes migrations, full resyncs and large file imports, and the responses your client has to handle for them +slug: /integrations/integration-toolkit/large-data-loads +--- + +# Large Data Loads + +An initial migration, a full resync or a large file import sends far more events than an integration's everyday traffic. The Integration Toolkit recognizes such a load by itself and processes it on capacity set aside for it, so that it does not slow down other organizations and is not slowed down by them. + +:::info What this means for you +- **Everyday traffic is processed promptly.** Another organization's large load does not slow it down. +- **Large loads need no configuration.** They are recognized automatically. Nothing is rejected in the normal case, nothing is lost, and the results are the same as for any other event. +- **Large loads take longer.** The first part is processed at once. After a short pause, the rest follows at a steady pace, which can take hours for a very large migration. +- **Interactive inbound is unaffected.** Pushes an end customer is waiting for keep their own lane. +- **Order is kept.** Events for the same entity are processed in the order in which they were accepted, also when a large load starts and when it ends. See [Order of Events](#order-of-events). +- **Two responses to handle.** Rarely, a request is answered with `503` and a `Retry-After` header: with the code `BULK_CAPACITY_EXHAUSTED` when the capacity for large loads is in use, or with `INGEST_THROTTLED` when ingestion is busy for a moment. Wait for the `Retry-After` interval and send the same request again. See [When Capacity Is Exhausted](#when-capacity-is-exhausted) and [When Ingestion Is Busy](#when-ingestion-is-busy). +- **A limit on the size of one event and one request.** An event may produce at most 5,000 updates, and a request at most 25,000. See [Size Limits](#size-limits). +- **You can book a window for a planned load.** In the Integration Hub or through the API, so that a migration runs on its own capacity from its first event. See [Bulk Import Windows](#bulk-import-windows). +::: + +## Everyday Traffic and Large Loads + +The inbound events of all organizations are processed together. Each organization has an **allowance** for that: how many events it can send in a short time and still count as everyday traffic. Single changes, delta syncs and occasional batches stay within it. + +An organization that sends more than its allowance covers is treated as running a **large load**. A migration or a full resync does that: it sends thousands of events within minutes. From then on, the organization's events are processed on capacity set aside for the load: + +- **At a pace that follows the health of the services behind it**, not as fast as the events arrive. A load moves faster when the platform is quiet and slower when it is busy, so that it does not degrade everyday traffic. +- **With the same ordering rules as everyday traffic.** What the inbound API says about the order of events, under `group_id` in the [API reference](https://docs.epilot.io/api/erp-integration), applies to a large load as well. The order is also kept between the events sent before the load was recognized and the load itself; see [Order of Events](#order-of-events). +- **Until the load has finished** and the organization has paused: for 15 minutes without new events, or for 3 minutes when the load has run for at least 30 minutes and only a trickle of events follows it (fewer than about one event every two seconds). After that, its events count as everyday traffic again. + +Your large load does not slow down other organizations, and their large loads do not slow down your everyday traffic. + +### What Counts {#what-counts} + +- Every event sent to the [inbound API](./inbound/getting-started.md), on every version of the endpoint. +- Every row of a [file import](#file-imports). +- Every [replayed](./monitoring/investigating.md#replaying-events) event. +- Every inbound event that a [managed call](./overview.md#managed-calls) produces. + +Events processed with [interactive inbound](./interactive-inbound.md) do not count. + +**Events that produce very many updates.** An event that turns into very many entity updates counts for more than one that writes a single entity. Its size is known only when it is processed, so it is counted then, not when it is received. If such an event takes your organization over its allowance, the request that carried it is not refused and is still processed as everyday traffic. The requests you send after it are processed as part of the large load. + +## What a Large Load Looks Like {#what-a-large-load-looks-like} + +From the outside, a large load runs in three steps: + +1. **The beginning is processed at once**, like everyday traffic, up to the allowance. +2. **A short pause.** Once the load is recognized, the rest of it waits until the events already queued ahead of it have been processed. This usually takes a few minutes. Your requests are accepted as usual during the pause. +3. **A steady pace.** The rest of the load is processed steadily until it is done. For a very large migration, this can take hours. + +The results are the same as for everyday traffic: the same mappings, the same entities and the same [monitoring codes](./monitoring/codes.md). Follow the progress in the [Monitoring tab](./monitoring/overview.md), where the events appear as they are processed. + +**Your own everyday traffic joins the load.** While a load runs, everything else the organization sends through the inbound API and file imports is processed as part of it, at the load's pace. A delta sync sent during a migration therefore takes longer than usual. Send what an end customer is waiting for with [interactive inbound](./interactive-inbound.md), which a large load does not delay. + +There are no fixed processing times. How long a load takes depends on its size, on the work each event causes and on how busy the platform is. + +### Order of Events {#order-of-events} + +Events for the same entity are processed in the order in which they were accepted. Moving an organization's events to the capacity for large loads does not change that: + +- **When a load starts.** The rest of the load waits until the events accepted before it have been processed. That is the short pause of step 2. +- **When a load ends.** The organization's events count as everyday traffic again only after the load has been processed completely. +- **If epilot switches this way of processing off** while a load runs, the load is finished where it is, and the events that follow it are processed after it. + +There is one exception. The pause of step 2 has a time limit of 30 minutes, and of 45 minutes for meter readings. If the events accepted before the load are still not processed by then, the load starts anyway, so that one stuck event cannot hold it back for ever. In that case an earlier event can be processed after a later one for the same entity. This is rare, and epilot is notified when it happens. + +## When Capacity Is Exhausted {#when-capacity-is-exhausted} + +The capacity for large loads that run at the same time is limited. If all of it is in use at the moment your organization goes over its allowance, the request that goes over it is refused. This is rare. Requests within the allowance are never refused for this reason, and neither are interactive events. + +Two more things are never refused for lack of capacity: + +- **Managed calls.** The call to the external system has already been made when its result is queued, so the inbound event of a [managed call](./overview.md#managed-calls) is always accepted. The same holds when the allowance cannot be checked. +- **File imports.** An import waits for capacity instead of failing; see [File Imports](#file-imports). + +### The Response + +A refused request is answered with HTTP `503`, the header `Retry-After: 60` and the error code `BULK_CAPACITY_EXHAUSTED`. This applies to `POST /v3/erp/updates/events`, to the deprecated v1 and v2 endpoints, and to [replays](./monitoring/investigating.md#replaying-events). + +```http title="503 — a request refused whole" +HTTP/1.1 503 Service Unavailable +Retry-After: 60 +Content-Type: application/json + +{ + "code": "BULK_CAPACITY_EXHAUSTED", + "message": "The capacity for large data loads is in use, and this organization is over its allowance for everyday traffic. Nothing of this request was accepted. Send the request again after the Retry-After interval." +} +``` + +The body is the error alone: it carries no `correlation_id` and no `results`. Nothing of the request was queued, and its deduplication ids were not used up, so the same request sent again is processed, not skipped as a duplicate. + +### What to Do {#what-to-do} + +1. **Wait for the `Retry-After` interval**, in seconds: 60 for `BULK_CAPACITY_EXHAUSTED`, 2 for [`INGEST_THROTTLED`](#when-ingestion-is-busy). Read it from the header rather than using a fixed delay. Retrying sooner does not help. +2. **Send the same request again**, unchanged. +3. **Hold back the requests that follow it** until the retry is accepted, so that your events keep the order in which you send them. +4. **If the retry is refused again, wait again.** Capacity becomes free as other loads finish. If refusals continue for a long time, contact epilot support. + +Give every event a `deduplication_id`. A request refused whole can be sent again either way, and the retry creates no duplicates, because nothing of it was accepted. With deduplication ids, every retry, whether after a `503`, a timeout or a lost response, is recognized and never processed twice. + +These steps apply to both `503` responses, `BULK_CAPACITY_EXHAUSTED` and `INGEST_THROTTLED`. + +### Requests That Mix Interactive and Other Events {#mixed-requests} + +Interactive events do not count towards the allowance and are never refused for lack of capacity. When a request carries interactive events and others, for example `interactive: true` on the request while one of its events belongs to a use case that does not allow interactive processing, only the other events are refused. The response is still a `503` with `Retry-After`, and it lists every event: + +```json title="503 — a mixed request: the interactive event was accepted" +{ + "code": "BULK_CAPACITY_EXHAUSTED", + "message": "Some events of this request were not accepted; see results", + "correlation_id": "018f8e9b-5a1b-7c4e-9b2a-4f0f6c1d7a21", + "results": [ + { + "event_id": "0199c3f2-8a41-7d0e-b6c7-2f1a9d3e4b55", + "status": "success", + "interactive": true, + "lane": "interactive" + }, + { + "event_id": "0199c3f2-8a42-7a10-9c3e-5b7d2e1f0a66", + "status": "error", + "message": "This event was not accepted: the capacity for large data loads is in use, and this organization is over its allowance for everyday traffic. Send the event again after the Retry-After interval." + } + ] +} +``` + +- **`results` says which events were accepted.** Events with `success` were accepted. Events with `error` and a message that starts with "This event was not accepted" were refused. +- **Retry the refused events** after the `Retry-After` interval. You can also send the whole request again if every event carries a `deduplication_id`: the interactive events that were accepted are then recognized as duplicates and come back `skipped`. +- **A close is not applied.** A `correlation_complete: true` on such a request is ignored with the reason `retry_requested`, as the response's `correlation_close` says. The retry closes the correlation; see [What Became of the Close](./interactive-inbound.md#what-became-of-the-close). + +Events that asked for interactive processing but run as everyday traffic, because the organization is over its [interactive rate limit](./interactive-inbound.md#rate-limits), count as everyday traffic here as well and can be refused like any other. + +## When Ingestion Is Busy {#when-ingestion-is-busy} + +A second `503` has nothing to do with the size of your allowance. It is answered with the header `Retry-After: 2` and the error code `INGEST_THROTTLED` in two cases: + +- The service that checks your organization's allowance is busy and cannot answer. The request is then not accepted, rather than accepted unchecked: accepting it unchecked could put the events of a running large load out of order. +- The queue that receives the events throttled the request. + +Both are transient. This response replaces the `500` with the message `Unknown API error` that the inbound endpoints returned when the queue throttled a request. + +```http title="503 — nothing of the request was accepted" +HTTP/1.1 503 Service Unavailable +Retry-After: 2 +Content-Type: application/json + +{ + "code": "INGEST_THROTTLED", + "message": "The service that checks the allowance of this organization for incoming events is busy and could not answer. Nothing of this request was accepted. Send the request again after the Retry-After interval." +} +``` + +When the body is `code` and `message` alone, nothing of the request was accepted or queued, and its deduplication ids were not used up. Send the same request again after the stated time. The retry is accepted and creates no duplicates. + +When some events of the request were accepted, the body also carries the message `Some events of this request were not accepted; see results`, the `correlation_id` and `results`, as for a [mixed request](#mixed-requests): + +- Events with `success` are queued. Do not send them again. +- Events with `error` were not accepted. Send them again after the `Retry-After` interval. +- With a `deduplication_id` on every event you can send the whole request again: the accepted events come back `skipped`. +- Events leave in the order of the request, and the send stops at the first event that is not accepted. An event that was not accepted is never followed by an accepted one on the same queue. +- A `correlation_complete: true` on such a request is not applied. The retry carries it again. + +This happens when the queue took a part of a large request before it throttled, and when the allowance could not be checked for a request that also carries interactive events, which are never counted and are accepted as usual. + +A [replay](./monitoring/investigating.md#replaying-events) is answered in the same way when nothing of it was queued. A replay that was throttled part of the way is answered with `200` and a `Retry-After` header; see [Replaying Events](./monitoring/investigating.md#replaying-events). + +## Size Limits {#size-limits} + +One event can turn into many entity updates, for example when its mapping creates one entity per item of a list in the payload. Two limits bound that: + +| Limit | Value | Response when exceeded | +|---|---|---| +| Updates one event may produce | 5,000 | `422`. The event's result has `status: "error"` and `code: "EVENT_TOO_WIDE"`. The other events of the request are processed. | +| Updates the events of one request may produce together | 25,000 | `422` with `code: "REQUEST_TOO_WIDE"`. The whole request is refused and nothing of it is queued. | + +An update is one entity that the event's mapping writes. An event that produces exactly 5,000 updates is accepted; 5,001 are refused. Meter readings are not counted, and file imports are not limited this way. + + +```json title="422 — one event of the request is too large" +{ + "correlation_id": "018f8e9b-5a1b-7c4e-9b2a-4f0f6c1d7a21", + "results": [ + { + "event_id": "0199c3f2-8a41-7d0e-b6c7-2f1a9d3e4b55", + "status": "success" + }, + { + "event_id": "0199c3f2-8a42-7a10-9c3e-5b7d2e1f0a66", + "status": "error", + "code": "EVENT_TOO_WIDE", + "message": "This event would produce 5,001 updates, more than the maximum of 5,000 updates per event. Split it into several events." + } + ] +} +``` + +```json title="422 — the request as a whole is too large" +{ + "code": "REQUEST_TOO_WIDE", + "message": "This request would produce at least 30,000 updates, more than the maximum of 25,000 updates per request (and 5,000 per event). Nothing of it was accepted. Split it into several requests." +} +``` + +What to do: + +- **`EVENT_TOO_WIDE`:** send the data of the event again as several smaller events. Nothing of the event was queued. The event also appears in monitoring with the code `EVENT_TOO_WIDE`. +- **`REQUEST_TOO_WIDE`:** send the events again in several smaller requests. The deduplication ids of the request were not used up. The count in the message is a lower bound, because the events after the one that brought the request over the limit are not counted. +- **Do not retry unchanged.** Unlike a `503`, a `422` for size is answered the same way every time. + +A result can also carry the code `EVENT_EVALUATION_LIMIT`: the mapping of the event ran for more than 25 seconds, or built a generated sequence of more than 4 million items, and was stopped before its updates could be counted. Nothing of the event was queued. Send its data in several smaller events. + +The inbound event of a managed call cannot be refused with a `422`, because the call has already answered. If it produces more than 5,000 updates, it is not processed, and it appears in monitoring with the code `EVENT_TOO_WIDE`. Change the call or its mapping and run the call again. + +## File Imports {#file-imports} + +A file import counts as a whole. A large import is recognized as a large load when it starts, and its rows are processed as described above. + +If the capacity for large loads is in use when an import needs it, the import does not fail. It waits and tries again by itself every 60 seconds, for up to 6 hours in total, and nothing needs to be done. It waits in the same way when the organization's allowance cannot be checked for a moment. + +- Its status stays `PROCESSING`. +- While it waits, its `progress` carries a `waiting_for_capacity` object. epilot's import screens show such an import as **Waiting for capacity**. +- Rows already sent stay sent. When capacity is free, the import continues where it stopped, and no row is sent twice. +- You can abort the import while it waits. + +```json title="GET /v2/erp/imports/{importId} — an import waiting for capacity (excerpt)" +{ + "import_id": "7c1e4b2a-93d5-4f08-b6a1-2e9d0c4f5a37", + "status": "PROCESSING", + "progress": { + "processed_rows": 0, + "total_rows": 250000, + "waiting_for_capacity": { + "since": "2026-11-16T08:12:04Z", + "retry_at": "2026-11-16T08:47:05Z", + "gives_up_at": "2026-11-16T14:12:04Z" + } + } +} +``` + +| Field | Meaning | +|---|---| +| `since` | When this wait began. It does not change while the same wait continues. | +| `retry_at` | When the import tries again, if known. The import tries again about every minute. | +| `gives_up_at` | The earliest time at which the import fails if capacity is still not free: the end of the 6 hours the import waits in total, counted across all of its waits. It can fail a little later, never earlier. | + +`waiting_for_capacity` is present only while the import waits. It is gone as soon as the import continues, when it is aborted, and when it ends. An import that never had to wait does not carry it. The field is the same on the single-import and list endpoints of both import APIs, `/v2/erp/imports` and `/v2/imports`. + +Only if no capacity becomes free within the 6 hours does the import fail, with the error code `BULK_CAPACITY_EXHAUSTED` and this message: + +```text +the capacity for large data loads stayed in use for the whole waiting time and the import was stopped. Rows already sent stay sent, please run the import again later +``` + +Run the import again later: validate it, then execute it. The rows that were already sent are sent again and update the same entities. + +## Bulk Import Windows {#bulk-import-windows} + +A large load is recognized by itself, but only after it has used up the organization's allowance: its first events are processed as everyday traffic, and the rest waits for the short pause of [step 2](#what-a-large-load-looks-like). If you know when a migration or a full resync will run, book a **bulk import window** for it. During the window, your organization's events run on their own capacity from the first event, and a pause between two batches of the load does not end it. + +A window applies to the whole organization, to every integration and file import, not to a single integration. + +### In the Integration Hub + +1. Open an integration and go to the **Monitoring** tab. +2. In the **Bulk import window** card, choose **Plan a bulk import**. +3. Pick the start date and time (in your time zone), the duration in hours, and add a note on what the load is, for example "Initial migration of all contracts". +4. Save. The card shows the window as **Scheduled**, and as **Active** while it runs. + +You can edit a window until it starts, and cancel it at any time. Cancelling an active window does not stop or slow down what is already running: the load continues and ends by the normal rules. + +While your organization's events run on their own capacity, whether in a window or because a load was recognized, the card says so, with the number of updates still waiting: + +- **Getting your lane ready:** earlier updates are finishing first; this is the short pause of step 2. +- **Your updates run on their own lane:** the load is being processed. + +Booking, editing and cancelling need the permission to manage integrations. Everyone who can view integrations sees the card. + +### Rules for a Window + +| Rule | Value | +|---|---| +| Length | at most 24 hours | +| Start | at most 30 days ahead, not in the past | +| Note | required, at most 500 characters | +| Windows per organization | one at a time; booking a new one replaces a window that has not started | + +The capacity for windows is shared by all organizations. If too many windows overlap, a booking is refused; choose another time, or contact epilot support for a very large migration. + +### Through the API + +The same window can be read, booked and cancelled with the Integration Toolkit API. The organization is the one of the token. + +```http title="Book a window" +PUT /v1/bulk-imports/window +Content-Type: application/json + +{ + "starts_at": "2026-11-24T06:00:00Z", + "ends_at": "2026-11-24T18:00:00Z", + "note": "Initial migration of all contracts" +} +``` + +- `GET /v1/bulk-imports/window` returns the window, if there is one, with its `status` (`scheduled`, `active` or `ended`), and the organization's current processing in `lane`. +- `DELETE /v1/bulk-imports/window` cancels the window. +- A refused booking is answered with `409` and an `error_code`, for example `window-active` (an active window cannot be replaced, only cancelled), `window-too-long` or `too-many-windows`. + +See the [API reference](https://docs.epilot.io/api/erp-integration) for the full schema. + +## Planning a Migration {#planning-a-migration} + +- **Book a bulk import window for the migration.** See [Bulk Import Windows](#bulk-import-windows). For an initial load of a whole customer base, also tell your epilot contact or epilot support ahead of time. +- **Verify the mapping on a subset first.** Use the [mapping simulation endpoint](./overview.md#inbound-api) and a small batch. A mapping error in a large load repeats for every event. +- **Handle both `503` responses before you start**, and give every event a `deduplication_id`. See [What to Do](#what-to-do). +- **Keep events and requests within the size limits.** At most 5,000 updates per event and 25,000 per request. See [Size Limits](#size-limits). +- **Plan for hours, not minutes.** Allow for a very large migration to take hours, and schedule cut-overs and go-live dates accordingly. +- **Keep end customers' flows on interactive inbound.** Registrations and contract additions that happen during a migration are not delayed by it. Do not set `interactive: true` on the migration itself; see [Common Mistakes](./interactive-inbound.md#common-mistakes). +- **Rehearse in a sandbox with a subset, and run the full volume in production.** Large loads in a sandbox are processed at a deliberately low rate; see [Sandbox Organizations](./sandbox-organizations.md). + +## Next Steps + +- [Sandbox Organizations](./sandbox-organizations.md) — how large loads behave in a sandbox +- [Inbound Getting Started](./inbound/getting-started.md) — the inbound API and its responses +- [Interactive Inbound](./interactive-inbound.md) — pushes an end customer is waiting for +- [Investigating Events](./monitoring/investigating.md) — traces and replay diff --git a/docs/integrations/integration-toolkit/monitoring/investigating.md b/docs/integrations/integration-toolkit/monitoring/investigating.md index a5bf188e..f090987f 100644 --- a/docs/integrations/integration-toolkit/monitoring/investigating.md +++ b/docs/integrations/integration-toolkit/monitoring/investigating.md @@ -103,10 +103,22 @@ Things worth knowing before you use it: - **14-day horizon.** The event is rebuilt from the stored inbound payload, which is kept for 14 days — see [retention](./overview.md#how-long-data-is-kept). Older events return `not_found`. -- **Always HTTP 200.** Per-event outcomes are in `results`, one entry per requested id - in request order, with a `status` of `success` / `queued`, `not_found`, `skipped`, - `ignored` or `error`. Compare `replayed` against the number of ids you sent rather - than trusting the status code. +- **HTTP 200, unless nothing was queued.** Per-event outcomes are in `results`, one entry per + requested id in request order, with a `status` of `success` / `queued`, `not_found`, + `skipped`, `ignored` or `error`. Compare `replayed` against the number of ids you sent + rather than trusting the status code. +- **A replay counts as inbound traffic.** If it takes your organization over its + allowance while the capacity for large loads is exhausted, the whole replay is + refused with `503`, a `Retry-After` header and `BULK_CAPACITY_EXHAUSTED`, and nothing + of it is queued. Send it again after the `Retry-After` interval — see + [Large Data Loads](../large-data-loads.md#when-capacity-is-exhausted). +- **A replay can be throttled.** If nothing of it was queued, it is refused with `503`, + `Retry-After: 2` and `INGEST_THROTTLED`, and you can send it again as it was — see + [When Ingestion Is Busy](../large-data-loads.md#when-ingestion-is-busy). If the queue + took a part of it, the answer is `200` with a `Retry-After` header: the events with + `success` are queued, the others have `error`. Replay only the events that are not + `success`. Replaying the whole request would queue the first ones a second time, + because every replay gets new deduplication ids. - **A replay is a new event.** It is assigned a fresh `event_id` and a new `correlation_id` prefixed `replay_`, so the replayed run is distinguishable from the original in monitoring rather than overwriting it. diff --git a/docs/integrations/integration-toolkit/overview.md b/docs/integrations/integration-toolkit/overview.md index 4705e3dc..c7799713 100644 --- a/docs/integrations/integration-toolkit/overview.md +++ b/docs/integrations/integration-toolkit/overview.md @@ -163,6 +163,8 @@ Push data from your ERP into epilot. Typical flows: Inbound ingestion is asynchronous. For the flows where an end customer is waiting for the push to land — portal self-registration, adding a contract — use [Interactive Inbound](./interactive-inbound.md): your middleware sends the portal's correlation id with the push, the events run on a dedicated lane, and the portal waits until exactly that data has been written. +Once the processing of large data loads is switched on, large loads, such as an initial migration or a full resync, are recognized automatically and processed on capacity set aside for them, so that they do not slow down the everyday traffic of other organizations. See [Large Data Loads](./large-data-loads.md) for what to expect and the responses your client has to handle, and [Sandbox Organizations](./sandbox-organizations.md) for how large loads behave in a sandbox. + ### Outbound (epilot to ERP) Push epilot events to your ERP via webhooks. Typical flows: diff --git a/docs/integrations/integration-toolkit/sandbox-organizations.md b/docs/integrations/integration-toolkit/sandbox-organizations.md new file mode 100644 index 00000000..b34f2414 --- /dev/null +++ b/docs/integrations/integration-toolkit/sandbox-organizations.md @@ -0,0 +1,47 @@ +--- +sidebar_position: 8 +title: Sandbox Organizations +description: How the Integration Toolkit processes the events of sandbox organizations, and how to plan large data loads in them +slug: /integrations/integration-toolkit/sandbox-organizations +--- + +# Sandbox Organizations + +A sandbox organization is where you build and verify an integration: its mappings, its use cases and the real-time flows your end customers will use. The Integration Toolkit processes a sandbox's everyday inbound traffic exactly as it does a production organization's. [Large data loads](./large-data-loads.md) are the difference: in a sandbox, they share a small capacity with the large loads of all other sandbox organizations and run at a deliberately low rate. + +:::info What this means for you +- **Everyday traffic is processed as in production.** Single events, small batches and test runs are as quick as in a production organization. +- **Large loads take much longer.** A full data load can take many times longer in a sandbox than in a production organization. +- **Nothing is lost.** Every event a sandbox accepts is processed, with the same results as in production. +- **Real-time flows are unaffected.** Events sent with [interactive inbound](./interactive-inbound.md) keep their own lane in a sandbox too. +- **A bulk import window works here too.** A [booked window](./large-data-loads.md#bulk-import-windows) puts a sandbox's load on the shared sandbox capacity from its first event; it still runs at the low rate. +::: + +## Which Organizations Are Sandboxes + +epilot sets an organization's type when it is created: a sandbox belongs to a production organization and is used to prepare changes before they go live. You do not configure anything in the Integration Toolkit. Every page of the Integration Hub shows a banner while you work in a sandbox organization. + +## What Is Different in a Sandbox + +Everything on [Large Data Loads](./large-data-loads.md) applies to sandboxes, with one difference: the large load of a sandbox runs on a small capacity that all sandbox organizations share, at a deliberately low rate, until it has finished. + +- **Everyday traffic**, within the organization's allowance, is processed together with every other organization's, as in production. +- **Large loads**, whether sent through the inbound API, as a file import or as a replay, run on the capacity all sandboxes share. +- **Interactive events** run exactly as in production, within the same [rate limits](./interactive-inbound.md#rate-limits), so you can test registrations and contract additions end to end. +- **The `503` responses** described in [When Capacity Is Exhausted](./large-data-loads.md#when-capacity-is-exhausted) and [When Ingestion Is Busy](./large-data-loads.md#when-ingestion-is-busy) can occur in a sandbox as well, and are handled the same way. The [size limits](./large-data-loads.md#size-limits) apply too. + +## How You Notice It + +- A banner on every page of the Integration Hub, linking to this page. +- The monitoring code [`SANDBOX_LANE`](./monitoring/codes.md) (level `info`) on the integration while the events of a large load arrive through the inbound API, at most once per hour per integration. +- The events of a large load are processed later than they would be in a production organization. Their results are the same. + +## Planning Large Data Loads + +- **Test with a representative subset.** A few hundred records that cover your edge cases verify a mapping as well as the full data set does, and they are processed quickly. +- **Run full-volume loads in production.** Initial migrations and full resyncs belong in the production organization. See [Planning a Migration](./large-data-loads.md#planning-a-migration). +- **Ask epilot ahead of a full-volume rehearsal.** If you need a full-volume dress rehearsal in a sandbox, for example before a go-live date, ask your epilot contact or epilot support in advance. For the agreed period, the sandbox is processed like a production organization. + +## Why Sandboxes Are Treated Differently + +Sandbox and production organizations share the same processing platform. Keeping the large loads of sandboxes on a small, shared capacity means that a test in a sandbox can never slow down a production organization, including your own.