Skip to content

fix(harvester): skip deleted files in incremental runs - #1234

Open
Yeagerist0 wants to merge 1 commit into
OWASP:mainfrom
Yeagerist0:fix/harvester-deleted-files
Open

Yeagerist0 wants to merge 1 commit into
OWASP:mainfrom
Yeagerist0:fix/harvester-deleted-files

Conversation

@Yeagerist0

Copy link
Copy Markdown

Summary

Fixes #1192. Incremental harvesting now excludes files deleted between the checkpoint and the target commit. Previously git diff --name-only returned deleted paths, and _harvest_repository() tried to read them from the target commit. That raised an error before the checkpoint could advance, leaving the repository stuck on every later run.

Use Git's --diff-filter=d so added, modified, and renamed files remain candidates while deleted paths are omitted. A real-repository integration test covers one edited file alongside one deleted file; the test failed before the change and passes now.

Verification

  • Harvester test suite: 129 passed, 2 skipped
  • Black check on all three changed files: passed
  • Focused mypy check on change_detector.py: passed
  • Full strict mypy reports 246 pre-existing errors in unrelated modules; no error in the changed file with imports silenced
  • Independent review: no blocker found

AI assistance was used to prepare the fix and regression test. I reviewed the diff and ran the checks locally.

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: OWASP/OpenCRE
  • Branch: main
  • File: .coderabbit.yml
  • Commit: 6bc7a30
  • Layers: Repository YAML
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7602d605-86c7-4bee-a38b-f987d6b2e13c
Commits

Reviewing files that changed from the base of the PR and between 6bc7a30 and 0cf4b03.

Files selected for processing: 3

  • application/tests/harvester_test/change_detector_test.py
  • application/tests/harvester_test/git_repository_client_integration_test.py
  • application/utils/harvester/change_detector.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harvester: a file deleted upstream permanently blocks incremental harvesting of that repository

1 participant