Skip to content

Commit

Permalink
Fix LLaMA model loading
Browse files Browse the repository at this point in the history
Fix hot model reload with `drama_llama` backend not working #7

* Loading is now entirely in the worker thread.
* During load, the settings are locked but the brief blocking before is gone.
* Several new `Request` and `Response` messages were added.

The issue with the panic for unsupported models (on Metal at least) is still there. That requires changes in `llama.cpp` itself or duplication of code.
  • Loading branch information
mdegans committed Jun 14, 2024
1 parent d75254a commit 77b9eda
Show file tree
Hide file tree
Showing 6 changed files with 846 additions and 305 deletions.
Loading

0 comments on commit 77b9eda

Please sign in to comment.