-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add markdown/notion importer #258
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add importer for my notion export - re-name prior importer client to sync client - add diff dependency for temporary hacky testing helper - allow creating documents without indexing, to support import-then-sync approach - types to es2020 for a misc lib function I needed
- add table to track import items and import links to support importing in steps - resolve notion note links while importing; if good update them to chronicles format so they work as note links very hacky / messy
- re-work tests to separate known issues with my current Notion export, vs hypothetical ones and non-Notion import issues - handle remaining cases - clean-up prasing related logging
cloverich
force-pushed
the
import_markdown
branch
from
October 14, 2024 22:15
dd5a55b
to
a01702e
Compare
- improve name-generation by using just folder name, and allowing root folder to be a journal name; slicing to handle length. Tons of edge cases but pretty decent overall
- now referenced files are imported to the _attachments directory - import_item status is now updated to help with debugging next steps are to debug some lingering failed imports (~5/200 notes), then track all links to confirm everything is imported (or not) as expected, then move on to clean-up
- fix some missing awaits that resulted in race conditions; all notes now import with note and file references - add basic status tracking on import items; very messy needs refactoring and cleanup
- if file link points to a url, dont try to import it (as a file); ignore instead - if file link has query params (e.g. ?size=800), strip them Fixes about 25 file import errors (all valid files now import)
- move faux tests into own routine, add button for it - move front-matter code into isolated module - light clean-up re-naming part 2 / ?
- use a staging table for file references when importing; its slower but easier to debug validate - remove import links table / tracking, unused at this point - some misc. clean-up (still v. messy, but improving)
- move all db code to knex - simplify note links updating - organize note links / file links logic so its grouped in importer - clean-up many comments - drop error tracking on import items except for final step
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an experimental Notion importer in support of #132 / #134. Once a Notion export is unzipped, use a new button in preferences screen to import the directory and referenced files:
The routine will import all markdown files, re-name and re-structure them to Chronicles format (one-level deep sub-folder for all notes, filename uuid), update note links, and import and update any referenced files
It's listed as experimental because I only tested it against my own export (~400 notes) and haven't worked through all edge cases. It uses staging tables (
imports
,import_notes
,import_files
) in a two-pass approach; the database entries can be used to understand what was imported (tracks source path and destination chronicles journal / id), but this is not exposed in the app (Its a SQLite database, easily viewed, see settings file). Once imported, the sync routine is called to synchronize the database with the new files.Changes:
Todo:
Tags
)Closes #134
Out of scope: