diff --git a/packages/server/README.md b/packages/server/README.md index 2f3bfd78..88f17816 100644 --- a/packages/server/README.md +++ b/packages/server/README.md @@ -222,23 +222,9 @@ const handler = createChatHandler({ ## Custom integrations -`createChatHandler(config)` returns `handleChat(body, clientIp, options)`, an async generator of SSE `data` payloads. The built-in adapters use it; call it directly for other runtimes. Pass `options.signal` and abort it when the visitor disconnects: the provider request is cancelled, nothing more is yielded, and the partial reply isn't saved. +For a runtime with the Fetch API's `Request` and `Response` (Next.js, Hono, Bun, Deno, Workers), pass requests to `chatcopsVercelHandler(config)`: it's a plain `(Request) => Promise`. For Node's `req` and `res` (`node:http`, Fastify, Koa), turn the request into a `Request` and stream the `Response` back. The docs have the code: [Next.js, Hono, Bun and Deno](https://chat.codercops.com/server/fetch-runtimes/). Route both the chat path and `/session` to it, since the widget gets its session token there. -```typescript -const abort = new AbortController(); -const stream = new ReadableStream({ - async start(controller) { - for await (const chunk of handler.handleChat(body, clientIp, { signal: abort.signal })) { - if (abort.signal.aborted) break; - controller.enqueue(new TextEncoder().encode(`data: ${chunk}\n\n`)); - } - if (!abort.signal.aborted) controller.close(); - }, - cancel() { - abort.abort(); // the runtime cancels the response when the client goes away - }, -}); -``` +The handler does CORS, `allowedOrigins`, body limits, pings, session tokens and the 0.5 format for old widgets, so bridging to it is better than calling `createChatHandler` and writing the HTTP side yourself. ## License diff --git a/website/src/content/docs/core/providers.mdx b/website/src/content/docs/core/providers.mdx index 6a190a4e..a2560e8e 100644 --- a/website/src/content/docs/core/providers.mdx +++ b/website/src/content/docs/core/providers.mdx @@ -232,6 +232,26 @@ for await (const part of turn) { Provider errors don't throw. They end the turn with an `error` part and a `finish` part with reason `error`. The built-in providers have only `step()`, so to get a reply from one outside the server, use `runTurn()`. It also gives you the reply's parts, status and token usage. (`chat()` and `chatSync()` on the built-in providers were removed in 1.0.) +### runTurn options + +| Option | What it's for | +|---|---| +| `provider`, `messages`, `systemPrompt`, `conversationId` | Required. `messages` is the history, oldest first, ending with the visitor's message | +| `systemContext` | Per-request instructions, sent after the system prompt so a cached prompt stays cached | +| `sources` | Retrieved chunks the reply can cite. Each cited page becomes a `source` part, once | +| `tools` | A `toolRegistry([...])`. Without it the model gets no tools | +| `messageId` | The id to save the reply under. A new one by default | +| `maxSteps` | Model requests in the turn, the final answer included. Default 6 | +| `toolTimeoutMs`, `maxTokens`, `temperature` | As on the server config | +| `signal` | Aborts the turn, for example when the visitor disconnects. The reply is saved as `aborted` | +| `exposeUsage` | Adds token usage to the `finish` part | +| `logger` | Where warnings and tool errors go. Default `consoleLogger()` | +| `budget` | `{ guard, context }`: checked before each model request and charged after it | +| `onPersist(message, { final })` | Saves the reply: after each tool round with `final: false`, and once at the end with `final: true` | +| `onPart(part)` | Called as each tool call, tool result and emitted part (a captured lead) is added | + +The turn's return value is `{ status, message, usage, finishReason, error? }`. `status` is `complete`, `incomplete` (an error, the token limit or the step limit) or `aborted`, and `error` says what went wrong for an incomplete turn that ended in one. + For tests, `@chatcops/core/testing/provider` has `scriptedProvider()`, which plays back a script of steps instead of calling a model. ## Tool support diff --git a/website/src/content/docs/core/testing.mdx b/website/src/content/docs/core/testing.mdx index 2ad402c3..4e9edfe4 100644 --- a/website/src/content/docs/core/testing.mdx +++ b/website/src/content/docs/core/testing.mdx @@ -15,6 +15,8 @@ It exits 0 when every test passed, 1 when some failed, and 2 when it couldn't ru `chatcops test` reads the `server` section of `chatcops.config.ts`. It's the same config you give the server: the provider, system prompt, tools and knowledge. +The command, its flags and `defineConfig` are `@chatcops/cli`'s API. The package exports more (the runner, the assertion and reporter registries) for its own use, and those can change in a minor release until the CLI reaches 1.0. + ```ts import { LeadCaptureTool } from '@chatcops/core'; import { defineConfig } from '@chatcops/cli'; diff --git a/website/src/content/docs/server/fetch-runtimes.mdx b/website/src/content/docs/server/fetch-runtimes.mdx index 32112c6b..e9755e96 100644 --- a/website/src/content/docs/server/fetch-runtimes.mdx +++ b/website/src/content/docs/server/fetch-runtimes.mdx @@ -134,6 +134,58 @@ Deno.serve({ port: 8000 }, (req, info) => { Run it with `deno run --allow-net --allow-env --allow-read --allow-sys server.ts`. Pin the version in the `npm:` specifier as above. Deno also skips npm versions published in the last 24 hours by default, so right after a ChatCops release it may resolve the previous version for a day. +## Node's http module, Fastify, Koa + +A server that gives you Node's `req` and `res` instead of a `Request` can still use the same handler: turn the request into a `Request`, and stream the `Response` back. This works for `node:http` as below, and for Fastify (`request.raw`, `reply.raw`) or Koa (`ctx.req`, `ctx.res`) the same way. + +```typescript +import { createServer } from 'node:http'; +import { Readable } from 'node:stream'; +import { chatcopsVercelHandler } from '@chatcops/server'; + +const chat = chatcopsVercelHandler({ + provider: { type: 'claude', apiKey: process.env.ANTHROPIC_API_KEY! }, + systemPrompt: 'You are a helpful assistant.', + sessionSecrets: [process.env.CHATCOPS_SESSION_SECRET!], + // Set below from the socket, so a visitor can't choose it. + clientIp: (req) => req.headers.get('x-client-ip'), +}); + +createServer(async (req, res) => { + const { pathname } = new URL(req.url ?? '/', 'http://localhost'); + // The chat, and the session token the widget asks for first. + if (pathname !== '/api/chat' && pathname !== '/api/chat/session') { + res.writeHead(404).end(); + return; + } + + const headers = new Headers(); + for (const [name, value] of Object.entries(req.headers)) { + if (typeof value === 'string') headers.set(name, value); + } + headers.set('x-client-ip', req.socket.remoteAddress ?? ''); + const request = new Request(`http://localhost${req.url}`, { + method: req.method, + headers, + body: req.method === 'POST' ? (Readable.toWeb(req) as ReadableStream) : undefined, + duplex: 'half', // Node needs this to send a streamed body + } as RequestInit); + + const response = await chat(request); + res.writeHead(response.status, Object.fromEntries(response.headers)); + if (!response.body) { + res.end(); + return; + } + // Stop the reply, and the model request behind it, when the visitor leaves. + const body = Readable.fromWeb(response.body as import('node:stream/web').ReadableStream); + res.on('close', () => body.destroy()); + body.pipe(res); +}).listen(3000); +``` + +The handler does everything else itself: CORS and `allowedOrigins`, body limits, pings that keep the stream open through a slow tool, session tokens, and the 0.5 format for old widgets. That's why it's better to bridge to it than to call `createChatHandler` and write the HTTP side yourself. + ## The visitor's IP Rate limits and the per-IP budget key on the visitor's IP. The handler only reads it from a header that the platform sets itself, because a visitor can send any `X-Forwarded-For` they like and get a fresh limit with every request.