Skip to content

fix: load cached pretrained models in offline mode - #2396

Open
emme1t wants to merge 2 commits into
abetlen:mainfrom
emme1t:fix/offline-pretrained-loading-1801
Open

emme1t wants to merge 2 commits into
abetlen:mainfrom
emme1t:fix/offline-pretrained-loading-1801

Conversation

@emme1t

@emme1t emme1t commented Nov 1, 2026

Copy link
Copy Markdown

Llama.from_pretrained() currently lists the remote repository before resolving a cached model, so HF_HUB_OFFLINE=1 fails even when the GGUF is already cached. In offline mode, list the local Hub snapshot or a populated local_dir, then reuse the existing filename/glob matching and download resolution. Online repository discovery stays unchanged.

Fixes #1801.

Validation on Windows x64, Python 3.12.13, with the pinned llama.cpp CPU build:

  • 19 targeted tests pass, including 11 new cases for cached files, nested paths, globs, shards, missing/ambiguous matches, local directories and online discovery. The final new test file gives 10 failed / 1 passed on the original code.
  • The 11 new cases also pass with huggingface-hub==0.23.0; the main run uses 2.4.0.
  • Real offline inference with ggml-org/models / tinyllamas/stories260K.gguf succeeds for both the literal filename and tinyllamas/stories260*.gguf. Both generate 8 tokens, , there was a little girl, from Once upon a time.
  • Repository-wide Ruff check and format check pass. The full multi-model test suite was not run locally.

The native build used LLVM-MinGW with Ninja, GGML_NATIVE=OFF, GGML_OPENMP=OFF, and _WIN32_WINNT=0x0A00. Offline matching considers files already present on disk. A requested local_dir must contain the model, following Hugging Face Hub's offline behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Llama.from_pretrained should work with HF_HUB_OFFLINE=1

1 participant