Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve sync and import performance #282

Open
cloverich opened this issue Dec 9, 2024 · 0 comments
Open

Improve sync and import performance #282

cloverich opened this issue Dec 9, 2024 · 0 comments

Comments

@cloverich
Copy link
Owner

Now that Chronicles stores note contents as markdown files, and indexes over them, the database is a cache that needs kept in sync; it has a sync routine for this (it is very similar to importer, so everything here applies to both). Its mostly naive and non-performant because its not important enough to do right now, but here's some ideas:

  1. Make the sync / import routine concurrent. Should be pretty straight forward. Likely would do w/ workers to make it easy to cap resource usage
  2. Some kind of content hashing to quickly determine if a file needs re-indexed or not
  3. (done) don't descend into directories needlessly
  4. allow configuring which directories and prefixes to ignore (or perhaps, which to sync)
  5. Do not delete indexed data when run starts (update during and after)
  6. Run in separate process
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant