Add video and audio transcription function for ingestion

[sm18lr88]

First, congrats on the YCombinator news!

My idea:
Include transcription option, so users can ingest video and audio knowledge. I have tons of knowledge only available on video that I have been transcribing for ingestion by Khoj.

I made my own Python app that can transcribe all the way from video -> audio -> text automatically with a GUI, but it's nowhere as robust and optimized as the Whisper.cpp project.

What do you think?

[debanjum]

First, congrats on the YCombinator news!

Thanks!

Include transcription option, so users can ingest video and audio knowledge.

Cool idea. Neat that you've built something for yourself already to allow using it with Khoj!
What are these videos about? Like is it personal recordings or talk/lectures you've got from the internet?

We could use whisper to allow folks to search, chat with audio, video stuff as well. But I'll have to think about how to prioritize this. The tradeoff between number of data sources we support and having Khoj work smoothly is real 😅

Can we also try make your video/audio to transcripts tool be an external plugin for Khoj? Khoj supports this since a few months now. A Khoj third party plugin just needs to output data into a structured jsonl file and then Khoj can index it for search and chat on it's own.

[sm18lr88]

Would be an honor to have my first app used.
I'm on a road trip right now but the latest version can even fetch YT videos for transcription. It's just the GUI is ridiculously unrefined, but it works, lol.

Let me know how I can help. I have a career-defining exam in the coming week so I'll be out of commission, but will make sure the app is current in the repository.

I currently use it for med school lectures, or to summarize YT channels with GPT-4 or Claude 2, like "CoinBureau" or "David Shapiro ~ AI".