Does autoLLM support local LLMs?

[jonny7737]

With few exceptions, most supported LLMs are 4K context window. Local LLMs at least give the opportunity for 16K, 32K and more.

[fcakyon]

Hello @jonny7737, OpenAI provides 16K context windiw models, clause-v2 has 100K context window.

Moreover, you can use any opensource HuggingFace and Ollama models with autollm (since we are using LiteLMM underhood). Feel free to ask more questions!

[jonny7737]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

[Meathelix1]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

They cant.. The only thing you are hosting locally would be the database that these models speak too, or the model from the Local LLM.

[fcakyon]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

They don't work locally, but they do have a larger context window.

You should use HuggignFace and Ollama models as local alternatives with autollm.

[igoralvarezz]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

They don't work locally, but they do have a larger context window.

You should use HuggignFace and Ollama models as local alternatives with autollm.

Is there a way to use local (offline) models from huggingface without the use of Ollama (currently not available for Windows OS yet)?

[fcakyon]

Yes @igoralvarezz it is currently supported. You can create an instance of HuggingFaceLLM and use it in the autollm pipeline as:

import torch

from llama_index.llms import HuggingFaceLLM
from autollm import AutoServiceContext, AutoVectorStoreIndex, AutoQueryEngine

llm = HuggingFaceLLM(
    context_window=4096,
    max_new_tokens=256,
    query_wrapper_prompt,
    tokenizer_name="StabilityAI/stablelm-tuned-alpha-3b",
    model_name="StabilityAI/stablelm-tuned-alpha-3b",
    device_map="auto",
    stopping_ids=[50278, 50279, 50277, 1, 0],
)

service_context = AutoServiceContext.from_defaults(llm=llm)
vector_store_index = AutoVectorStoreIndex.from_defaults()
query_engine = AutoQueryEngine.from_instances(vector_store_index. service_context)

[Dipeshpal]

Can we load llm using langchain?
If not, then please integrate langchain.

I am trying langchain because it supports local models.
Ways of loading models with langchain: https://github.com/Dipeshpal/langchain_tutorial/blob/main/load_llm_models.py

Example-

from langchain.llms import OpenAI
from autollm import AutoServiceContext, AutoVectorStoreIndex, AutoQueryEngine, read_files_as_documents
import os

os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxx'


def load_openai_llm_model():
    llm = OpenAI()
    return llm


documents = read_files_as_documents(input_dir="d2")
llm = load_openai_llm_model()
service_context = AutoServiceContext.from_defaults(llm=llm)
vector_store_index = AutoVectorStoreIndex.from_defaults(documents=documents)
query_engine = AutoQueryEngine.from_instances(vector_store_index, service_context)

response = query_engine.query("What is LLAMA index")
print(response.response)

BTW, this script gives me below error-

D:\Projects\tmp\venv\Scripts\python.exe D:\Projects\tmp\auto.py 
2023-11-07 01:26:13,888 - autollm - INFO - Reading files from d2..
2023-11-07 01:26:13,888 - autollm - INFO - Found 1 'document(s)'.
Parsing documents into nodes: 100%|██████████| 1/1 [00:00<00:00, 247.60it/s]
Generating embeddings: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s]
Traceback (most recent call last):
  File "D:\Projects\tmp\auto.py", line 19, in <module>
    response = query_engine.query("What is LLAMA index")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\indices\query\base.py", line 31, in query
    return self._query(str_or_query_bundle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\query_engine\retriever_query_engine.py", line 182, in _query
    response = self._response_synthesizer.synthesize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\response_synthesizers\base.py", line 147, in synthesize
    response_str = self.get_response(
                   ^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\response_synthesizers\compact_and_refine.py", line 38, in get_response
    return super().get_response(
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\response_synthesizers\refine.py", line 127, in get_response
    response = self._give_response_single(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\response_synthesizers\refine.py", line 182, in _give_response_single
    program(
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\response_synthesizers\refine.py", line 53, in __call__
    answer = self._llm_predictor.predict(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\llm_predictor\base.py", line 195, in predict
    formatted_prompt = self._extend_prompt(formatted_prompt)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\llm_predictor\base.py", line 287, in _extend_prompt
    extended_prompt = self.query_wrapper_prompt.format(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\tmp\venv\Lib\site-packages\llama_index\prompts\base.py", line 157, in format
    prompt = self.template.format(**mapped_all_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'context_str'

Process finished with exit code 1

[Dipeshpal]

I tried LangChainLLM from llama_index.llms also but still same error-

from llama_index.llms import LangChainLLM
from langchain.llms import OpenAI

def load_openai_llm_model():
    llm = LangChainLLM(llm=OpenAI())
    return llm


documents = read_files_as_documents(input_dir="d2")
llm = load_openai_llm_model()
service_context = AutoServiceContext.from_defaults(llm=llm)
vector_store_index = AutoVectorStoreIndex.from_defaults(documents=documents)
query_engine = AutoQueryEngine.from_instances(vector_store_index, service_context)

response = query_engine.query("What is LLAMA index")
print(response.response)