Skip to content

AI copilot that sits in the call (paid tier, free models first) - #164

Merged
ralyodio merged 2 commits into
masterfrom
feat/meeting-copilot
Oct 28, 2026
Merged

ralyodio merged 2 commits into
masterfrom
feat/meeting-copilot

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

An optional in-call AI agent for one owner, on web, desktop, CLI, API and MCP. PRD: plans/MEETING-COPILOT.md.

What it does (Roofle parity)

  • Role presets: Sales, Recruiter, Product Manager, Founder, Customer Success, Engineer, General.
  • Live transcript tagged by speaker; suggested next question with the reason and a jump to the line it came from.
  • Rolling notes (summary, key points, decisions, action items), open questions kept until answered, key figures and dates.
  • "Recap the last 30 s", private "ask the copilot", one-click follow-ups (action items + recap message), and a conversation review (focus area, talk/listen, questions answered, follow-up clarity) that says when the transcript is too thin.
  • Everyone in the meeting sees "AI copilot is listening for "; everything else is private to the owner.

Paid tier, free models first

  • Pro/Team (or admin) only.
  • Default path is free and self-hosted: Whisper (speaches, faster-whisper-small) + qwen2.5:7b on Ollama, both on dev2 in the pairux-copilot compose project.
  • Paid providers are the fallback or override: pay as you go at provider cost + 10% from a prepaid CoinPay wallet ($5/$20/$50), or the owner's own Anthropic/OpenAI/ElevenLabs keys (AES-256-GCM at rest, last 4 shown). Modes: auto, free, byok, cloud.

How

  • Audio reuses the translation uploads: the copilot is an invisible listener in its own live-translate RoomTranslator; /translation/feed feeds both rooms and listening counts both, so existing web/desktop clients feed it unchanged.
  • Small local models get a bounded JSON schema (grammar-constrained decoding must close fields) and all output passes guards.
  • New package @pairux/meeting-copilot (types, roles, hook, shared CopilotPanel for web + desktop). API: /api/sessions/:id/copilot{,/stream,/ask,/recap,/questions}, /api/copilot{,/:id,/:id/follow-ups,/:id/review,/settings,/wallet}, /api/billing/copilot-topup, MCP at /api/v1/mcp. Pages: /copilot, /copilot/:id, /settings/copilot. CLI: pairux copilot status|start|stop|watch|notes|ask|recap|followups|review|list|settings|wallet.
  • Migration 20261028120000_meeting_copilot (dry-run on prod in a rolled-back transaction: OK). Needs applying by hand on dev2.

Verified

  • Web: tsc 0 errors, next build OK, full vitest 114 files / 1209 tests pass (32 new copilot tests), i18n parity 38/38 across 19 locales.
  • Package tests 5/5; desktop tsc 0; agent-cli copilot tests pass.
  • Free path end to end against the real dev2 containers: STT ~2.2 s per phrase; notes update 37 s; follow-ups + review 20 s.
  • Known: agent-cli/auth.test.ts loopback tests fail on dev2 locally with code identical to master (green in master CI); leaving CI to arbitrate.

🤖 Generated with Claude Code

An optional in-call AI agent for one owner, on web, desktop, CLI, API and MCP
(plans/MEETING-COPILOT.md). Roofle parity: role presets, live transcript by
speaker, suggested next question with jump-to-line, rolling notes (summary,
key points, decisions, action items), open questions tracked until answered,
key figures and dates, 30-second recap, private "ask the copilot", one-click
follow-ups and a conversation review that says when the call is too thin.
Everyone in the meeting sees "AI copilot is listening for <name>".

- Pro/Team (or admin) only. Free self-hosted open-source models are the
  default path (Whisper via speaches, qwen2.5 via Ollama); paid providers are
  the fallback or override: pay as you go at provider cost + 10% from a
  prepaid CoinPay wallet, or the owner's own Anthropic/OpenAI/ElevenLabs keys
  (AES-256-GCM at rest). Modes: auto, free, byok, cloud.
- Audio rides the existing translation uploads: the copilot is an invisible
  listener in its own live-translate RoomTranslator, so current clients feed it.
- Local models get a bounded JSON schema and every provider's output passes
  guards (one question per suggestion, length caps, no invented answers).
- New: @pairux/meeting-copilot (types, roles, hook, shared panel), API under
  /api/sessions/:id/copilot and /api/copilot, /api/v1/mcp, /copilot pages,
  /settings/copilot, `pairux copilot ...` CLI, migration
  20261028120000_meeting_copilot, strings in all 19 locales.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
body: JSON.stringify({ topup: id }),
})
);
if (data.checkout_url) window.location.href = data.checkout_url;
async function signIn(extra: Partial<CliConfig> = {}) {
await saveConfig(
{
tokens: { accessToken: 'pux_at_x', refreshToken: 'pux_rt_x', expiresAt: Date.now() + 3e6 },
@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

52 finding(s)

HIGH/CRITICAL: 4 | MEDIUM: 29 | LOW: 19

Severity Rule Location
HIGH sh-eval-expansion .githooks/pre-commit:33
HIGH js-electron-node-integration apps/desktop/src/main/window.ts:49
HIGH sh-unquoted-expansion-destructive apps/installer/scripts/install.sh:717
HIGH sh-unquoted-expansion-destructive apps/installer/scripts/install.sh:921
MEDIUM sh-remote-script-execution apps/installer/scripts/install.sh:693
MEDIUM sh-remote-script-execution apps/installer/scripts/install.sh:824
MEDIUM sh-remote-script-execution apps/installer/scripts/install.sh:826
MEDIUM sh-remote-script-execution apps/installer/scripts/install.sh:1084
MEDIUM sh-remote-script-execution apps/installer/scripts/install.sh:1086
MEDIUM sh-remote-script-execution apps/livekit/setup-livekit-server.sh:93
MEDIUM sh-remote-script-execution apps/turn/deploy-droplet.sh:62
MEDIUM js-unescaped-html-sink apps/web/src/app/blog/[slug]/page.tsx:52
MEDIUM js-unescaped-html-sink apps/web/src/app/blog/[slug]/page.tsx:79
MEDIUM js-unescaped-html-sink apps/web/src/app/c/[handle]/page.tsx:199
MEDIUM js-open-redirect apps/web/src/app/cli/authorize/consent.tsx:68
MEDIUM js-unescaped-html-sink apps/web/src/app/l/[joinCode]/page.tsx:146
MEDIUM js-unescaped-html-sink apps/web/src/app/l/[joinCode]/page.tsx:230
MEDIUM js-unescaped-html-sink apps/web/src/app/layout.tsx:150
MEDIUM js-unescaped-html-sink apps/web/src/app/live/page.tsx:154
MEDIUM js-unescaped-html-sink apps/web/src/app/page.tsx:98
MEDIUM js-unescaped-html-sink apps/web/src/app/pricing/page.tsx:262
MEDIUM js-open-redirect apps/web/src/app/settings/copilot/copilot-settings.tsx:105
MEDIUM js-unescaped-html-sink apps/web/src/app/u/[username]/page.tsx:285
MEDIUM js-open-redirect apps/web/src/components/billing/useCheckout.ts:81
MEDIUM js-unescaped-html-sink apps/web/src/components/pit-footer.tsx:24
MEDIUM js-open-redirect apps/web/src/hooks/useDesktopHandoff.ts:24
MEDIUM redos-nested-quantifier apps/web/src/lib/deliverable.ts:11
MEDIUM js-unescaped-html-sink apps/web/src/lib/player/player.ts:124
MEDIUM js-unescaped-html-sink apps/web/src/lib/player/player.ts:393
MEDIUM js-unescaped-html-sink apps/web/src/lib/player/player.ts:396
MEDIUM js-unescaped-html-sink apps/web/src/lib/player/player.ts:405
MEDIUM js-unescaped-html-sink apps/web/src/lib/player/player.ts:608
MEDIUM sql-template-interpolation packages/ai-core/src/prompts.ts:36
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:138
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:139
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:340
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:341
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:371
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:372
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:383
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:397
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:398
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:405
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:415
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:416
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:453
LOW secret-generic-credential apps/desktop/src/main/agent-cli/commands.test.ts:454
LOW secret-generic-credential apps/desktop/src/main/agent-cli/copilot.test.ts:92
LOW secret-generic-credential apps/livekit/fly.toml:12
LOW secret-generic-credential apps/turn/fly.toml:11

…and 2 more. Full results in the Security tab.

Snippets are redacted; ThreatCrush never prints matched credential material.

…m-no-tag-length)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ralyodio
ralyodio merged commit 0fbcebe into master Oct 28, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants