diff --git a/reference/database/schema.md b/reference/database/schema.md index a3a54753a..7035be5bf 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -381,7 +381,7 @@ The attribute type must be `[Float]`. The attribute is automatically indexed wit Write semantics: -- Creating a record with the source field, or updating the source field, computes the vector before the write commits (with `inputType: 'document'`). A failure to compute the embedding fails the write. +- Creating a record with the source field, or updating the source field, computes the vector before the write commits (with `inputType: 'document'`). A failure to compute the embedding fails the write. That includes a source value longer than the model's context window, on a provider that rejects over-long inputs. `@embed` does not pass [`truncate: true`](../models/api#truncating-long-inputs); only `models.embed()` calls can set it. - An update that does not touch the source field leaves the vector unchanged. - Setting the source field to `null` sets the vector to `null`. - Replicated writes and audit-log replays do not re-embed — the vector travels with the record, and only the node that accepted the original write calls the model. diff --git a/reference/models/api.md b/reference/models/api.md index 8f9e51a74..1c0d51a72 100644 --- a/reference/models/api.md +++ b/reference/models/api.md @@ -3,7 +3,7 @@ id: api title: API --- - + @@ -24,12 +24,58 @@ const [single] = await models.embed('What is Harper?', { inputType: 'query' }); const batch = await models.embed(['first document', 'second document']); ``` -| Option | Type | Default | Description | -| ----------- | ------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- | -| `model` | `string` | `'default'` | Logical name of a configured embedding model | -| `requires` | `Capability[]` | — | Capabilities the chosen backend must satisfy; used by [routing](./routing#capability-routing) to select a candidate in the group | -| `inputType` | `'document'` \| `'query'` | — | Hint for models that distinguish document embeddings from query embeddings (e.g. `nomic-embed-text`); ignored by models that do not | -| `signal` | `AbortSignal` | — | Cancels the call; composed with the backend's configured `requestTimeoutMs` | +| Option | Type | Default | Description | +| ----------- | ------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `model` | `string` | `'default'` | Logical name of a configured embedding model | +| `requires` | `Capability[]` | — | Capabilities the chosen backend must satisfy; used by [routing](./routing#capability-routing) to select a candidate in the group | +| `inputType` | `'document'` \| `'query'` | — | Hint for models that distinguish document embeddings from query embeddings (e.g. `nomic-embed-text`); ignored by models that do not | +| `truncate` | `boolean` | `false` | From v5.4.0, cut each input to what the model accepts instead of failing on an over-long input — see [Truncating long inputs](#truncating-long-inputs) | +| `signal` | `AbortSignal` | — | Cancels the call; composed with the backend's configured `requestTimeoutMs` | + +### Truncating long inputs + + + +An input longer than the embedding model's context window fails by default on most providers. With `truncate: true`, each input is cut to what the model accepts. The cut keeps the start of the text, so any task prefix (such as `search_document: `) stays in. Each input still produces one vector, embedded from that start only; the rest of the text does not contribute to it. Most backends send the full input and ask the provider to cut it, so the request is no smaller; for Bedrock Cohere embed v3, Harper itself first cuts each text to 2,048 characters (see the table below). + +`truncate: true` requires the backend's `truncate` capability, in the same way that declaring tools requires `tools`. [Routing](./routing#capability-routing) skips any backend in the group that cannot truncate, fallback entries included. If no candidate can, the call fails with a capability error (`ModelCapabilityError`) before any request is sent. A backend that can truncate can still fail after the request, like any other call. + +`requires: ['truncate']` only selects a backend that can truncate; it does not turn truncation on. Pass `truncate: true` for that. + +When `truncate` is absent or `false`, Harper sends the same request as before v5.4.0, and the provider's own handling applies. `false` does not mean "never cut": a provider that cuts by default still does. + +| Backend | `truncate: true` | `truncate` absent or `false` | +| ----------------------------------------------------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `openai` with [`truncation: vllm`](./backends#openai) | Sends `truncate_prompt_tokens: -1` and `truncation_side: 'right'` (vLLM 0.18 or later) | vLLM rejects an over-long input | +| `openai` without `truncation` | Not capable; skipped | api.openai.com rejects an over-long input; other OpenAI-compatible servers apply their own default | +| `ollama` | Sends `truncate: true` on `/api/embed` | Ollama cuts to the loaded context by default | +| `bedrock`, Cohere embed v4 | Sends `truncate: 'RIGHT'` | Bedrock's default for the model | +| `bedrock`, Cohere embed v3 | Cuts each text to 2,048 characters, then sends `truncate: 'END'` | Cohere cuts tokens by default, but Bedrock rejects a text over 2,048 characters | +| `bedrock`, Titan | Not capable; skipped | Titan rejects an over-long input | + +See [Backends](./backends) for each backend's details, and [`defineBackend()`](./backends#definebackend) for declaring the capability on a custom backend. + +```yaml +# harper-config.yaml +models: + embedding: + default: + backend: openai + apiKey: ${VLLM_API_KEY} + baseUrl: http://vllm.internal:8000/v1 + model: BAAI/bge-m3 + truncation: vllm +``` + +```javascript +import { models } from 'harper'; + +const [vector] = await models.embed(longText, { truncate: true }); +``` + +Turning truncation on does not re-embed records that are already stored. Vectors written earlier stay as they are, and an input whose embedding failed earlier is embedded only when it is written again. + +The [`@embed`](../database/schema#embed) directive and the OpenAI-compatible `/v1/embeddings` endpoint do not expose this option yet; only `models.embed()` calls can set it ([HarperFast/harper#3705](https://github.com/HarperFast/harper/issues/3705)). ## generate() @@ -290,7 +336,7 @@ Registers a custom backend under a logical name, selectable by the `model` optio ## Errors and timeouts - An unconfigured logical model name throws a not-found error. The error names the missing logical name only — it does not enumerate configured names. -- A capability mismatch (embedding call to a generation-only backend, tool declarations against a backend without tool support, `requires: ['calibrated']` against an uncalibrated decision backend) throws before any request is made. A decision backend that reports a single call as uncalibrated when `calibrated` was required fails that attempt after the request, and the next candidate is tried. +- A capability mismatch (embedding call to a generation-only backend, tool declarations against a backend without tool support, `truncate: true` against a backend that cannot truncate, `requires: ['calibrated']` against an uncalibrated decision backend) throws before any request is made. A decision backend that reports a single call as uncalibrated when `calibrated` was required fails that attempt after the request, and the next candidate is tried. - A malformed decision schema or state rejects with a `400` error before any request is made, and is not recorded. - On a read-only node, `recordOutcome()` rejects with a `503` because it is a write, and so does `decide()` with `persist: true`, before any request is made; a `decide()` without it works there. A `decide()` with `persist: true` whose record cannot be committed after the backend answered rejects with a `500` without trying another candidate. - `recordOutcome()` rejects with a `404` for an id that does not exist, has expired, or has not replicated to this node yet, and with a `400` for a report that does not fit the stored schema. diff --git a/reference/models/backends.md b/reference/models/backends.md index be4b71886..481c88fe1 100644 --- a/reference/models/backends.md +++ b/reference/models/backends.md @@ -53,6 +53,8 @@ models: When embedding with `nomic-embed-text`, the `inputType` option (`'document'` / `'query'`) is applied using the model's task prefixes; other models ignore it. +From v5.4.0, the Ollama backend has the `truncate` capability: an `embed()` call with [`truncate: true`](./api#truncating-long-inputs) sends `truncate: true` on `/api/embed`. Ollama cuts each input to the loaded context even without it, so leaving the option unset changes nothing. + The Ollama backend does not advertise tool support — declaring tools against it fails up front. ## OpenAI @@ -78,15 +80,31 @@ models: model: meta-llama/Llama-3.1-8B-Instruct ``` -| Field | Default | Description | -| -------------- | --------------------------- | ---------------------------------------------------------------------------------- | -| `apiKey` | — (required) | API key, sent as a bearer token. Use `${VAR}` indirection | -| `baseUrl` | `https://api.openai.com/v1` | API root; point at any OpenAI-compatible endpoint | -| `model` | — | Model name, e.g. `gpt-4o`, `text-embedding-3-large` | -| `organization` | — | Sent as the `OpenAI-Organization` header, for keys spanning multiple organizations | +| Field | Default | Description | +| -------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------- | +| `apiKey` | — (required) | API key, sent as a bearer token. Use `${VAR}` indirection | +| `baseUrl` | `https://api.openai.com/v1` | API root; point at any OpenAI-compatible endpoint | +| `model` | — | Model name, e.g. `gpt-4o`, `text-embedding-3-large` | +| `organization` | — | Sent as the `OpenAI-Organization` header, for keys spanning multiple organizations | +| `truncation` | — | From v5.4.0, `vllm` on an embedding entry served by vLLM 0.18 or later, so `embed()` can honor `truncate: true` | `responseFormat: 'json'` maps to OpenAI's JSON mode and `responseFormat: { schema }` to strict structured outputs (`json_schema`); OpenAI-compatible servers vary in their support for these. + + +`truncation: vllm` lets an embedding entry serve [`embed()` calls with `truncate: true`](./api#truncating-long-inputs). It declares that the endpoint is vLLM 0.18 or later; Harper does not check. With it, a `truncate: true` call sends `truncate_prompt_tokens: -1` (the model's own input limit) and `truncation_side: 'right'` (keep the start). An older vLLM ignores `truncation_side` and may cut the start of the input instead. Without `truncation`, the entry does not advertise the `truncate` capability and never sends those fields, because api.openai.com rejects fields it does not know. `vllm` is the only accepted value, and configuration validation rejects `truncation` on generative entries and on other backends. + +```yaml +models: + embedding: + default: + backend: openai + apiKey: ${VLLM_API_KEY} + baseUrl: http://vllm.internal:8000/v1 + model: BAAI/bge-m3 + truncation: vllm +``` + ## Anthropic Calls the Anthropic Messages API. Generation only — Anthropic does not offer an embeddings API. @@ -139,6 +157,13 @@ models: The model identifier's vendor prefix (`anthropic.`, `meta.`, `amazon.titan-`, `cohere.`, `mistral.`) determines the request/response format Harper uses; an unrecognized prefix is rejected with an error. Tool support depends on the underlying model family. Bedrock embedding APIs accept one text per request, so batch `embed()` calls are issued sequentially. +From v5.4.0, an embedding entry whose `model` is Cohere embed v3 (`cohere.embed-english-v3`, `cohere.embed-multilingual-v3`, and their `-light` variants) or Cohere embed v4 (`cohere.embed-v4`) has the `truncate` capability, for [`embed()` calls with `truncate: true`](./api#truncating-long-inputs): + +- Cohere embed v4: Harper sends `truncate: 'RIGHT'`. +- Cohere embed v3: Harper first cuts each text to 2,048 characters (never splitting a surrogate pair), then sends `truncate: 'END'`. The character cut is needed because Bedrock rejects a text over 2,048 characters before any token truncation. Short prose then uses less than the model's 512-token window. Without `truncate: true`, Harper does not cut: Cohere v3 cuts tokens by default, but Bedrock rejects a text over 2,048 characters. + +Titan embedding models have no truncation setting and do not advertise the capability. + ## Generative decision adapter @@ -228,7 +253,7 @@ Custom backends can be added two ways: **registered programmatically** (below), models.defineBackend(spec: DefineBackendSpec): ModelBackend ``` -A method on `models` (reachable as `models.defineBackend(...)` / `scope.models.defineBackend(...)`). Builds a `ModelBackend` from the methods it implements. `capabilities()` is derived from which of `embed` / `generate` / `generateStream` / `decide` / `scoreChoices` are supplied; `tools`, `adapters`, `calibrated`, `structuredOutput`, `noMatch` and `calibratedNoMatch` cannot be inferred from method presence, so declare them explicitly; a backend that leaves them out is never routed calls that require them. `maxScoredChoices` is a limit rather than a capability: leave it out when `scoreChoices` accepts any number of choices. +A method on `models` (reachable as `models.defineBackend(...)` / `scope.models.defineBackend(...)`). Builds a `ModelBackend` from the methods it implements. `capabilities()` is derived from which of `embed` / `generate` / `generateStream` / `decide` / `scoreChoices` are supplied; `tools`, `adapters`, `calibrated`, `structuredOutput`, `noMatch`, `calibratedNoMatch` and `truncate` cannot be inferred from method presence, so declare them explicitly; a backend that leaves them out is never routed calls that require them. `maxScoredChoices` is a limit rather than a capability: leave it out when `scoreChoices` accepts any number of choices. | Field | Type | Default | Description | | ------------------- | ---------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -245,9 +270,12 @@ A method on `models` (reachable as `models.defineBackend(...)` / `scope.models.d | `structuredOutput` | `boolean` | `false` | Whether `generate` sends `responseFormat: { schema }` as a decoding constraint; selectable with `requires: ['structuredOutput']` | | `noMatch` | `boolean` | `false` | Whether `decide` returns a no-match score for leaves that set `noMatch: true`; required by such a call | | `calibratedNoMatch` | `boolean` | `false` | Whether that no-match score is calibrated; required with `calibrated` by an opted-in call that requires `calibrated` | +| `truncate` | `boolean` | `false` | From v5.4.0, whether `embed` honors `truncate: true`; required by such a call, and ignored unless `embed` is supplied | `embed`, `generate`, and `decide` return the shape the built-in backends return: `{ status: 'completed', output, usage? }`, where `output` is `Float32Array[]` for `embed`, `{ content, finishReason }` for `generate`, and a distribution (or per-field distributions) for `decide`. `generateStream` is an async generator yielding incremental `{ deltaContent?, deltaToolCalls?, finishReason? }` chunks — the same [`generateStream()`](./api#generatestream) shape, not a wrapped result. At least one method must be supplied. A backend that supplies only `generateStream` still satisfies `generate()`: Harper drains the stream into a single result. +`truncate: true` only declares the capability; Harper does not cut the input for you. An `embed` implementation declared with it must read `opts.truncate` and, when it is `true`, cut each input to what its model accepts, keeping the start, instead of failing. + ### registerBackend() diff --git a/reference/models/routing.md b/reference/models/routing.md index 92c149a49..31b7bd7f0 100644 --- a/reference/models/routing.md +++ b/reference/models/routing.md @@ -35,8 +35,9 @@ A call to the `default` model tries `openai` first; if it fails, the call falls A call can require capabilities of the backend it lands on. The router keeps only the candidates whose `capabilities()` satisfy the requirement, in group order. -- **`opts.requires`**: an explicit list of capabilities (`embed`, `generate`, `stream`, `tools`, `adapters`, `decide`, `calibrated`, `scoreChoices`, `structuredOutput`, `noMatch`, `calibratedNoMatch`). For example, `models.decide(state, schema, { requires: ['calibrated'] })` routes to a decision backend that reports calibrated probabilities, and the [generative decision adapter](./backends#generative-decision-adapter) routes its scoring calls on `scoreChoices`. +- **`opts.requires`**: an explicit list of capabilities (`embed`, `generate`, `stream`, `tools`, `adapters`, `decide`, `calibrated`, `scoreChoices`, `structuredOutput`, `noMatch`, `calibratedNoMatch`, and from v5.4.0 `truncate`). For example, `models.decide(state, schema, { requires: ['calibrated'] })` routes to a decision backend that reports calibrated probabilities, and the [generative decision adapter](./backends#generative-decision-adapter) routes its scoring calls on `scoreChoices`. - **Tools auto-require `tools`**: a `generate()` call whose input carries a `tools` array routes to a tools-capable candidate in the group instead of erroring on a backend that can't do tools. +- **`truncate: true` auto-requires `truncate`**: from v5.4.0, an `embed()` call with [`truncate: true`](./api#truncating-long-inputs) routes only to a candidate that can cut over-long inputs. ```javascript // Tools in the input auto-route to a tools-capable candidate in the group: diff --git a/release-notes/v5-lincoln/5.4.md b/release-notes/v5-lincoln/5.4.md index b9dc0e30f..7e48e6a99 100644 --- a/release-notes/v5-lincoln/5.4.md +++ b/release-notes/v5-lincoln/5.4.md @@ -76,6 +76,10 @@ The agent records the usage of each model call on its session, and `get_agent_se A `models.generative` entry can now declare `pricing: { inputPerMtok, outputPerMtok }`, what you pay in USD per million prompt and completion tokens. Harper ships no prices of its own. Each `generate()` call is priced at the rates of the entry that served it, a fallback included, and the result carries `costUsd` when that entry is priced and the provider reported both token counts. With `toolMode: 'auto'` it covers the final round, and streaming calls are not priced. This makes `maxCostUsd` work on `toolMode: 'auto'` calls; before, it had no prices behind it and never tripped. A `toolMode: 'auto'` round that leaves `maxCostUsd` or `maxToolTokens` unmeasurable, because the entry is unpriced or the provider reported no token counts, now stops the call with the budget error (429) instead of warning once and continuing, so set `pricing` on entries you cap by cost. See [Pricing](/reference/v5/models/overview#pricing). +### Truncating Long Embedding Inputs + +`models.embed(input, { truncate: true })` cuts each input to what the embedding model accepts, keeping the start of the text and any task prefix, instead of failing on an input longer than the model's context window. Only a backend that can truncate serves such a call: Ollama, Bedrock Cohere embed v3 and v4, and OpenAI-compatible embedding entries that set the new `truncation: vllm` for a vLLM 0.18 or later endpoint. If no candidate can, the call fails before any request is sent. Calls without the option send the same requests as before. Custom backends declare the capability with `defineBackend({ truncate: true })`. The `@embed` directive and the `/v1/embeddings` endpoint do not expose the option yet. See [Truncating long inputs](/reference/v5/models/api#truncating-long-inputs). + ## Memory ### Smaller Per-Thread Baseline