can anyone provide sample configuration files to enable searching markdown files #213

parimalpuranik · 2023-04-17T16:08:30Z

parimalpuranik
Apr 17, 2023

I want to enable khoj to search Obsidian markdown files and I tried to configure the UI using *.md ( I didn't know better) and when I tried to run khoj from command line I ran into following error:

PS C:\Users%USER%> C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe
--- Logging error ---
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\logging_init_.py", line 1101, in emit
stream.write(msg + self.terminator)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f318' in position 0: character maps to
Call stack:
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
sys.exit(run())
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 64, in run
logger.info("🌘 Starting Khoj")
Message: '🌘 Starting Khoj'
Arguments: ()
[08:34:12] INFO 🌘 Starting Khoj main.py:64
--- Logging error ---
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\logging_init.py", line 1101, in emit
stream.write(msg + self.terminator)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f4ac' in position 0: character maps to
Call stack:
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
sys.exit(run())
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 88, in run
configure_server(args, required=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 44, in configure_server
state.processor_config = configure_processor(args.config.processor)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 169, in configure_processor
logger.info("💬 Setting up conversation processor")
Message: '💬 Setting up conversation processor'
Arguments: ()
INFO 💬 Setting up conversation processor configure.py:169
--- Logging error ---
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\logging_init.py", line 1101, in emit
stream.write(msg + self.terminator)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f48e' in position 0: character maps to
Call stack:
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
sys.exit(run())
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 88, in run
configure_server(args, required=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 49, in configure_server
state.model = configure_search(state.model, state.config, args.regenerate)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 112, in configure_search
logger.info("💎 Setting up search for markdown notes")
Message: '💎 Setting up search for markdown notes'
Arguments: ()
INFO 💎 Setting up search for markdown notes configure.py:112
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe_main.py", line 7, in
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 88, in run
configure_server(args, required=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 49, in configure_server
state.model = configure_search(state.model, state.config, args.regenerate)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 114, in configure_search
model.markdown_search = text_search.setup(
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\search_type\text_search.py", line 180, in setup
entries_with_indices = text_to_jsonl(config).process(previous_entries)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\processor\markdown\markdown_to_jsonl.py", line 41, in process
MarkdownToJsonl.extract_markdown_entries(markdown_files)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\processor\markdown\markdown_to_jsonl.py", line 110, in extract_markdown_entries
with open(markdown_file, "r", encoding="utf8") as f:
OSError: [Errno 22] Invalid argument: 'C:\Users\%USER%\.md'
PS C:\Users%USER%>

I even tried autoconfigure from obsidian but even that didnt work. Perhaps if I had a sample configuration file to go by I could try using that

debanjum · 2023-04-17T17:34:25Z

debanjum
Apr 17, 2023
Maintainer

Hey @parimalpuranik, I've just fixed these issues today and released khoj version 0.6.1 with these fixes.

You should:

Update your khoj server with pip install --upgrade khoj-assistant and restart it (with khoj --no-gui)
Update your khoj obsidian plugin (from within Obsidian Settings > Community plugins > Khoj > Update). This should auto-configure your khoj server
Wait for your notes to get indexed by the server and now try Khoj search (and/or Khoj chat) from within Obsidian

Let me know if that fixes the issues you mentioned?

As for a sample configs, see khoj_docker.yml or khoj_sample.yml

1 reply

debanjum Apr 17, 2023
Maintainer

Also if you've messed around with your khoj configuration (and it's in a bad state), delete the khoj.yml file at C:\Users<your-windows-username>.khoj\khoj.yml`. Then restart the khoj server and the re-install/re-enable the khoj obsidian plugin to auto-configure khoj server

parimalpuranik · 2023-04-17T19:51:41Z

parimalpuranik
Apr 17, 2023
Author

I deleted the khoj.yml file and reinstalled / re-enabled the upgraded 0.6.1 plugin and restarted the khoj server. I had the auto-configure option enabled and indexed the notes and I think I have restarted the server a couple of times but I dont see a khoj.yml file in the directory you mention.

PS C:\Users%USER%> C:\Users%USER%\AppData\Local\Programs\Python\Python310\Scripts\khoj.exe --no-gui
[12:43:49] INFO 🌘 Starting Khoj main.py:64 WARNING Khoj is not configured. configure.py:36 Configure it via khoj GUI, plugins or by editing C:\Users%USER%.khoj\khoj.yml.
INFO 🌖 Khoj is ready to use main.py:135[12:43:50] INFO Started server process [11164] server.py:75 INFO Waiting for application startup. on.py:45 INFO Application startup complete. on.py:59 INFO Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) server.py:206

When I tried to run a search I got the familiar check khoj backend is running message.

1 reply

parimalpuranik Apr 17, 2023
Author

Some more error messages :

Exception in thread Thread-38:
Traceback (most recent call last):
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1009, in bootstrap_inner
self.run()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1358, in run
self.function(*self.args, **self.kwargs)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\main.py", line 147, in poll_task_scheduler
schedule.run_pending()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init.py", line 780, in run_pending
default_scheduler.run_pending()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init_.py", line 100, in run_pending
self.run_job(job)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init.py", line 172, in run_job
ret = job.run()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\schedule_init.py", line 661, in run
ret = self.job_func()
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 68, in update_search_index
state.model = configure_search(state.model, state.config, regenerate=False)
File "C:\Users%USER%\AppData\Local\Programs\Python\Python310\lib\site-packages\khoj\configure.py", line 87, in configure_search
if (t == state.SearchType.Org or t == None) and config.content_type.org:
AttributeError: 'NoneType' object has no attribute 'content_type'

parimalpuranik · 2023-04-24T13:36:09Z

parimalpuranik
Apr 24, 2023
Author

Thanks for your help, I have khoj working on my setup now. I will try running the chat part next.

3 replies

debanjum Apr 24, 2023
Maintainer

Ah, that's great! Do you know what you did to fix it?

Not sure why you saw the khoj backend not running error.
But you'd have to re-enable/re-install khoj obsidian (or just obsidian) to auto-configure khoj backend.
You'd have seen the AttributeError: 'NoneType' object has no attribute 'content_type' when you'd have tried to search as the khoj backend wasn't configured or hadn't finished indexing your notes

parimalpuranik Apr 25, 2023
Author

I think re-enabling / reinstalling khoj obsidian did the trick but I don't remember the full sequence of actions I have taken.

debanjum Apr 25, 2023
Maintainer

Ah, that's unfortunate but not an issue. Great you've got Khoj search finally working. Hopefully getting Khoj chat to work shouldn't be too problematic 🤞🏾

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can anyone provide sample configuration files to enable searching markdown files #213

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

can anyone provide sample configuration files to enable searching markdown files #213

parimalpuranik Apr 17, 2023

Replies: 3 comments · 5 replies

debanjum Apr 17, 2023 Maintainer

debanjum Apr 17, 2023 Maintainer

parimalpuranik Apr 17, 2023 Author

parimalpuranik Apr 17, 2023 Author

parimalpuranik Apr 24, 2023 Author

debanjum Apr 24, 2023 Maintainer

parimalpuranik Apr 25, 2023 Author

debanjum Apr 25, 2023 Maintainer

parimalpuranik
Apr 17, 2023

Replies: 3 comments 5 replies

debanjum
Apr 17, 2023
Maintainer

debanjum Apr 17, 2023
Maintainer

parimalpuranik
Apr 17, 2023
Author

parimalpuranik Apr 17, 2023
Author

parimalpuranik
Apr 24, 2023
Author

debanjum Apr 24, 2023
Maintainer

parimalpuranik Apr 25, 2023
Author

debanjum Apr 25, 2023
Maintainer