[themarolt]

Summary

Runs Octolens keyword-to-project mapping discovery on an hourly connectors_worker Temporal schedule instead of as a connector sync unit, and makes it create the integrations Octolens runs under: one per mapped subproject, plus one segment-less integration that owns the single mention-poll sync unit (Octolens's 500 requests/hour budget is shared org-wide). The mention poll already emits under the subproject integrations (#4988). Once this is deployed with CROWD_OCTOLENS_API_TOKEN set, Octolens mentions start arriving as activities.

Changes

• New octolensDiscovery workflow, runOctolensDiscovery activity and connectors-octolens-discovery schedule (hourly, overlap SKIP, registered in main.ts next to the dispatcher and shadow-diff schedules). Each run:
  • skips with an info log when CROWD_OCTOLENS_API_TOKEN is not set, before any write;
  • finds or creates the segment-less integration and its mentionPoll sync unit with emitEnabled: true (insert-only, so mentions become activities rather than shadow records);
  • runs the existing discovery rules unchanged (an incomplete Octolens snapshot still writes nothing);
  • in one transaction: finds, creates or restores the Octolens integration of each mapped subproject, upserts its mappings, deactivates mappings no longer seen, and soft-deletes subproject integrations left without keywords;
  • revives the mention-poll unit if it was dead-lettered, since only auth failures dead-letter a unit and discovery's own Octolens calls just succeeded with the same token.
• An auth failure is not retried within a run; other failures get one retry, and the next hourly run covers the rest.
• mappingDiscovery is removed from the Octolens manifest and becomes a plain function (octolens/mappingDiscovery.ts). Octolens deps keep only the keyword lookup the poll needs.
• Data access: new reviveDeadLetteredSyncUnits; the per-integration mapping queries that no longer have callers are removed.
• Tests: the activity (token guard, emitEnabled, call order inside the transaction, revive only after Octolens answered), the revive query (DB), and the moved discovery logic.

Deployment notes:

• Set CROWD_OCTOLENS_API_TOKEN on connectors_worker; without it the schedule runs and skips.
• The schedule is only created when missing. A later change to its interval or timeout needs the existing Temporal schedule deleted first.
• Prod has no Octolens sync units or integrations today; a hand-made mappingDiscovery unit would fail every hour, so check other environments with SELECT id, "integrationId", "syncName", status FROM integration.sync_units WHERE platform = 'octolens' before deploying there.
• Protected path: the data-access layer.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

Related issue

Refs linuxfoundation/lfx-cdp-insights-ops#748

[cursor]

PR Summary

Medium Risk
Writes integrations, keyword mappings, and sync-unit state on a schedule; incorrect concurrency or reconciliation could misroute mentions or soft-delete integrations, though transactions and SKIP overlap mitigate that.

Overview
Octolens keyword-to-project discovery now runs on an hourly Temporal schedule in connectors_worker instead of as a mappingDiscovery connector sync. The Octolens manifest keeps only mentionPoll; discovery logic is a standalone function that takes a reconcileMappings callback.

Each scheduled run no-ops when CROWD_OCTOLENS_API_TOKEN is unset. Otherwise it ensures a segment-less anchor integration and a single mentionPoll sync unit with emitEnabled: true on insert, runs the same Octolens API mapping rules (still skipping all DB writes on incomplete snapshots), then reconciles mappings in one transaction: create/restore one Octolens integration per subproject, upsert keywords, deactivate unseen mappings org-wide, and soft-delete empty subproject integrations. After a successful Octolens call it revives a dead-lettered mentionPoll on the anchor. The schedule uses overlap SKIP so concurrent discovery cannot cross-soft-delete integrations.

Supporting changes: isOctolensCredentialConfigured, reviveDeadLetteredSyncUnits, reconcileOctolensKeywordMappings in the data layer; Octolens deps trimmed to keyword routing only; tests for the activity, reconcile transaction, and revive behavior.

^{Reviewed by Cursor Bugbot for commit b37bdef. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🔵 Needs a closer look

Live ingestion and automatic integration restoration/deletion warrant human validation, particularly of transactional reconciliation.

1 open finding

• Use real database tests to verify reconciliation transactions and rollback · New

What changed in this PR

Moves Octolens mapping discovery to an hourly Temporal schedule, enabling mention ingestion through per-subproject integrations and one shared poll.

Changes:

• Registers discovery scheduling, workflow, and activity.
• Reconciles mappings transactionally and revives dead-lettered polling.
• Removes obsolete discovery dependencies and updates tests.

File	Description
services/​libs/​data-access-layer/​src/​octolens/​index.ts	Removes unused mapping queries.
services/​libs/​data-access-layer/​src/​octolens/​index.test.ts	Adapts mapping tests.
services/​libs/​data-access-layer/​src/​connectors/​syncUnits.ts	Adds dead-letter revival.
services/​libs/​data-access-layer/​src/​connectors/​syncUnits.test.ts	Tests revival scope and watermark preservation.
services/​libs/​connectors/​src/​credentials.ts	Adds token configuration check.
services/​libs/​connectors/​src/​connectors/​octolens/​syncs/​mentionPoll.test.ts	Removes obsolete dependency mocks.
services/​libs/​connectors/​src/​connectors/​octolens/​mappingDiscovery.ts	Separates discovery from connector syncs.
services/​libs/​connectors/​src/​connectors/​octolens/​mappingDiscovery.test.ts	Updates discovery tests.
services/​libs/​connectors/​src/​connectors/​octolens/​index.ts	Removes discovery from manifest.
services/​libs/​connectors/​src/​connectors/​octolens/​deps.ts	Separates project resolution from polling dependencies.
services/​libs/​connectors/​src/​connectors/​octolens/​deps.test.ts	Updates dependency tests.
services/​apps/​connectors_worker/​src/​workflows/​octolensDiscovery.ts	Defines discovery workflow and retries.
services/​apps/​connectors_worker/​src/​workflows.ts	Exports discovery workflow.
services/​apps/​connectors_worker/​src/​schedules/​octolensDiscovery.ts	Creates hourly, non-overlapping schedule.
services/​apps/​connectors_worker/​src/​main.ts	Registers discovery schedule.
services/​apps/​connectors_worker/​src/​activities/​octolensDiscoveryActivities.ts	Provisions polling and reconciles integrations.
services/​apps/​connectors_worker/​src/​activities/​octolensDiscoveryActivities.test.ts	Tests discovery orchestration and guards.
services/​apps/​connectors_worker/​src/​activities.ts	Exports discovery activity.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

Automated integration lifecycle changes and first-time live mention emission warrant code-owner review and database-test validation.

0 open findings

1 resolved since last review

• Use real database tests to verify reconciliation transactions and rollback

🧠 Review effort: Balanced