feat: make incognito work #44
Conversation
| messages=[ | ||
| {"content": FILE_PROMPT, "role": "system"}, | ||
| {"content": json.dumps(summaries), "role": "user"}, | ||
| {"content": WATCH_PROMPT, "role": "system"}, | ||
| {"content": json.dumps(fs_events), "role": "user"}, | ||
| ], | ||
| model="llama3-70b-8192", |
There was a problem hiding this comment.
I don't like this global variable, because I think it can switch incognito mode on/off for different requests or paths. But I don't want to learn about the watch handler global function whatever, and definitely don't plan on testing it.
src/loader.py
Outdated
| @@ -152,7 +156,6 @@ async def summarize_image_document(doc: ImageDocument, client): | |||
| client = ollama.AsyncClient() | |||
There was a problem hiding this comment.
I wanted to switch this to litellm as well, but they don't make it easy to send a local file for vision because of a dependency on requests https://github.com/BerriAI/litellm/blob/3a35a58859a145a4a568548316a1930340e7440a/litellm/llms/prompt_templates/factory.py#L624-L635
|
I still haven't gotten it to work consistently, so I changed it to a draft PR. |
| reader = SimpleDirectoryReader(input_files=[path]).iter_data() | ||
|
|
||
| docs = next(reader) | ||
| splitter = TokenTextSplitter(chunk_size=6144) | ||
| text = splitter.split_text("\n".join([d.text for d in docs]))[0] | ||
| doc = Document(text=text, metadata=docs[0].metadata) | ||
| summary = dispatch_summarize_document_sync(doc, client) |
There was a problem hiding this comment.
all these sync methods should come back for watch_util; or be replaced by a sync wrapper around the async ones (like https://stackoverflow.com/a/62949043 ) (I got rid of these out of naivety and a dislike of duplication)
|
Ironically, my manual tests using groq finish in time, but my manual tests with ollama are too slow to finish, or it's not using a capable enough model on my machine. |
refactor: remove ollama direct usage in favor of litellm direct usage bug: didn't work on the full sample data, kept looping on the pdf
closes #24
Implements incognito mode, and removes the groq api key floating around various places. Aggresively defaults to localai in order to avoid sending people's files to groq accidentally.
I think it's a breaking change to require ollama, which is actually needed to support image categorization.
P.S. I'm having trouble with create_file_tree, I can't figure out how to pass the incognito request to that function.
Also I did not test this outside the /batch endpoint from the api. So someone else should test the electron app / watch util if they care about it.