[jimdowling]

Summary

• New guide user_guides/fs/data_source/creation/sharepoint.md: the Entra ID application and its permissions, the cluster variables, creating a data source and a sync in the UI, the data source page, how a sync runs (job, editable script, read-only dataset, CSV audit log with a DuckDB example, failures, deletion), and the Python client and CLI. Added to the nav.
• The strict build passes for this page; the build's remaining warnings are agent pages that need a newer hopsworks-api than this ticket's branch.
• File index: the guide documents the index option, its columns and rules, compaction, the Python and CLI calls, and a DuckDB query from a terminal.

Related PRs (merge in this order)

1. logicalclocks/docker-images#1138
2. logicalclocks/hopsworks-ee#3573
3. logicalclocks/hopsworks-helm#2627
4. [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset hopsworks-api#1353
5. logicalclocks/hopsworks-front#2201
6. logicalclocks/loadtest#1186
7. [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset #715

Feature

HWORKS-3562: a SharePoint Online data source that copies a folder of a site's document library into a project dataset, once or on a schedule (one-way mirror). Each sync is an ordinary Python job in the python-feature-pipeline environment running DataSources/<connector>/sharepoint_sync.py (rclone), which data owners can edit. Every run that changes something writes a CSV audit file in Hive-style date= partitions, rotated after data_source_audit_retention_days. Failure alerts go to an optional alert receiver. Two-way sync and delegated (user) sign-in are out of scope.

Merge order: docker-images (image with rclone and duckdb) → hopsworks-ee → hopsworks-helm → hopsworks-api → hopsworks-front → loadtest → docs.

Testing

On cluster jim-sharepoint (5.3.0-SNAPSHOT) against a real Microsoft 365 tenant (site hopsworks-sync-test, app with Sites.ReadWrite.All), with the backend, frontend and SDK from these branches and the python-feature-pipeline image built from the docker-images branch:

• loadtest workflows/feature_store/python_driver/test_data_sources_sharepoint.py: 2 passed in 7m05s (2026-10-26, at these heads)
  • copy once: copies a folder subtree, audits added files and folders, a second run after a deletion in SharePoint deletes nothing; turning indexing on later indexes every copied file, and the copy keeps the row of a file it keeps;
  • scheduled mirror: initial copy with indexing, an idle run writes no audit file, then update, rename, delete and add in SharePoint are mirrored with matching audit and index rows; deleting the synced dataset is refused (110057) while a data owner may delete a file in it, and the next run restores it; a feature group the sync did not create cannot become its index (create and rename refused) and survives deleting the sync with its files; pause, then delete without keeping files removes the dataset and the sync's own index.
  • A first run of the same pair failed once on an Arrow Flight read deadline while Payara was recovering from a failed EAR copy; the rerun passed.
• Scheduled runs: two demo syncs ran on their schedules for a day, 80 and 22 runs, all succeeded, submitted on the quarter hour and the hour by the scheduler.
• Project storage quota: a sync into a project whose HopsFS quota was 250 MB above its usage failed the files that did not fit with disk quota exceeded; the run is partial with the project's storage quota is full and what to do about it. Its job failure alert reached a webhook alert receiver (alertmanager JobExecution, status Failed) 97 s after the run.
• Migration and Helm seeding: the packaged V128 migration, applied to a scratch schema on the cluster's RonDB, gives tables identical to the cluster's (SHOW CREATE TABLE); the chart's rendered DML for the nine variables, applied to a scratch table, matches the cluster's rows.
• File index: compaction checked by replaying the Delta log after two commits on an earlier day (OPTIMIZE, 4 files to 2, one per day); a 269-file folder re-ingested with indexing (269 rows) and queried with the DuckDB CLI in a terminal whose image adds this branch's DuckDB layer and SDK, at /hopsfs/featurestore/<project>_featurestore.db/<fg>_1.
• terminal-server DuckDB layer: built with the pinned CLI and delta extension; LOAD delta passes the signature check with the extension installed from the verified file.
• UI walk-through in a browser: data source form, test connection (toast), sync setup with the live library and folder tree, schedule presets, Syncs table and audit card with the DuckDB snippet, Code tab, file browser (sync icon, write actions hidden), Jobs badge, delete dialogs.
• Unit tests: the sync script's pytest suite (49, with rclone 1.75.2, duckdb 2.0.0 and deltalake 1.4.2, as in CI); ee SharePoint Java tests; api SDK and CLI tests (424); front vitest for the storage-connector pages.
• Reviews: Codex pre-reviews and adversarial rounds on the lifecycle, fencing and file index code, every finding verified and fixed or recorded; review comments from dhananjay-mk addressed (index ownership, failed replacement, quota-exceeded path, DuckDB checksums). Copilot could not review (quota).

Not verified on a cluster:

• a real helm upgrade running the packaged Flyway migration and DML (both checked against the cluster's database as above);
• Sites.Selected with a site-level grant (the test app has Sites.ReadWrite.All).

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.