Support for new Notion URL format #69

bhchiang · 2021-07-21T06:13:02Z

Not 100% sure, but I believe the URL format for Notion shared pages recently changed.

It's now notion.site instead of notion.so:

Editing view: https://www.notion.so/bryanchiang/Bryan-Chiang-fc01c67a1ed9402e83eb8efd5c99a216
Shared view: https://bryanchiang.notion.site/Bryan-Chiang-fc01c67a1ed9402e83eb8efd5c99a216

I get a parser error with the second one.

Ito-MacBook:loconotion bryanhpchiang$ python3 loconotion https://www.notion.so/bryanchiang/fc01c67a1ed9402e83eb8efd5c99a216
[23:09:54] INFO Initialising parser with simple page url
[23:09:54] INFO Setting output path to 'dist/bryanchiang/fc01c67a1ed9402e83eb8efd5c99a216'
[23:09:54] INFO Initialising chromedriver at /usr/local/lib/python3.9/site-packages/chromedriver_autoinstaller/91/chromedriver
[23:09:56] INFO Parsing page 'https://www.notion.so/bryanchiang/fc01c67a1ed9402e83eb8efd5c99a216'
[23:10:57] CRITICAL Timeout waiting for page content to load, or no content found. Are you sure the page is set to public?
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/bryanhpchiang/Documents/Workspace/loconotion/loconotion/__main__.py", line 144, in <module>
    main()
  File "/Users/bryanhpchiang/Documents/Workspace/loconotion/loconotion/__main__.py", line 123, in main
    Parser(config=config, args=vars(args))
  File "/Users/bryanhpchiang/Documents/Workspace/loconotion/loconotion/notionparser.py", line 85, in __init__
    self.run(url)
  File "/Users/bryanhpchiang/Documents/Workspace/loconotion/loconotion/notionparser.py", line 667, in run
    f"Finished!\n\nProcessed {len(tot_processed_pages)} pages in {formatted_time}"
TypeError: object of type 'NoneType' has no len()

Will trying modifying the check for a valid notion.so website.

The text was updated successfully, but these errors were encountered:

tomreitz · 2021-07-22T20:46:30Z

@bryanhpchiang it looks like the page you linked isn't publicly shared... unless you recently un-shared it, that would explain why loconotion cannot load the content.

There's definitely an issue here though, since shared pages have the new Notion URL format of https://example.notion.site/Page-1F29BC48EA1A029FC481B but sub-pages still have the old format https://notion.so/example/Page-1F29BC48EA1A029FC481B which is a redirect page.

For me, loconotion correctly loads the primary public Notion page URL, but times out on any subpages. I think the logic at lines 582-584 of loconotion/notionparser.py needs to be updated to rewrite page URLs from the old format to the new one before attempting to fetch them.

leoncvlt · 2021-07-22T21:23:50Z

@tomreitz merged a pull request from @bryanhpchiang earlier today which should address this, want to pull it and check it's all good?

tomreitz · 2021-07-22T21:53:27Z

@leoncvlt thanks for the quick response (and an awesome project!). Subpages still not working for me, see this public page which converts fine, but the subpages in the table time out, per the logs below

[21:47:02] INFO Initialising parser with configuration file
[21:47:02] INFO Setting output path to 'dist/wiwebsites.com'
[21:47:02] INFO Initialising chromedriver at /usr/bin/chromedriver
[21:47:03] INFO Parsing page 'https://tomreitz.notion.site/Wisconsin-Websites-ecdb3dc4cd1e40f280b7512a23ca2006'
[21:47:17] INFO Downloading 'https://www.notion.so/print.b31f28aa.css'
[21:47:17] INFO Downloading 'https://www.notion.so/app-7d82edb35207a8a8b776.css'
[21:47:18] INFO Downloading 'https://www.notion.so/lyon-text-regular-3be84b20b1d9ff1e3456b0a220ae449b.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/lyon-text-regular-italic-437d32a42fc5b8268bb4a1e0cc8b363f.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/lyon-text-semibold-acb7f110189034ff6a1afa4b730be0ed.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/lyon-text-semibold-italic-1f81a2f93060f05edd7f078ac91f25e6.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/iawriter-mono-regular-4b73d071988a4f1cd2283524716ad970.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/iawriter-mono-italic-d5d3224c1377168e261efc6aa0ce89c6.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/iawriter-mono-bold-eb96a5e539892d26cf8b0cb2367e3580.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/iawriter-mono-bold-italic-743b231fa82483406c79a00fa1f12fe8.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/inter-ui-regular-3ae6a7d3890c33d857fc00bd2e4c4820.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/inter-ui-medium-95b8a98959d1af9ab432d7ffe295ef94.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/inter-ui-semibold-19b57197b819695d334b9961ee41910e.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/inter-ui-bold-001893789f7f342b520f29ac8af7d6ca.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/permanent-marker-a6d62939e7c920a184ddddcf4149e62c.woff'
[21:47:18] INFO Downloading 'https://www.notion.so/katex/katex.88defe76.min.css'
[21:47:18] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_AMS-Regular.342a61e0.ttf'
[21:47:18] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Caligraphic-Bold.b27e354b.ttf'
[21:47:18] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Caligraphic-Regular.bd18bae2.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Fraktur-Bold.359e1e97.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Fraktur-Regular.6b53a2db.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Main-Bold.ed829b5f.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Main-BoldItalic.ca23ba4b.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Main-Italic.14ff9c98.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Main-Regular.c89c6436.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Math-BoldItalic.7b481bb8.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Math-Italic.f677173e.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_SansSerif-Bold.362d94c6.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_SansSerif-Italic.2c742978.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_SansSerif-Regular.6087fc04.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Script-Regular.781730b2.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Size1-Regular.54a80b37.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Size2-Regular.24cbe093.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Size3-Regular.ee3e5bf4.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Size4-Regular.b78c75bb.ttf'
[21:47:19] INFO Downloading 'https://www.notion.so/katex/fonts/KaTeX_Typewriter-Regular.90f78c10.ttf'
[21:47:19] INFO Exporting page 'https://tomreitz.notion.site/Wisconsin-Websites-ecdb3dc4cd1e40f280b7512a23ca2006' as 'index.html'
[21:47:19] INFO Parsing page 'https://www.notion.so/7514e88c4042418997665b5ecf11733b?v=703812ea01fe4ee6bc010fd72be278f8'
[21:48:20] CRITICAL Timeout waiting for page content to load, or no content found. Are you sure the page is set to public?
[21:48:20] INFO Parsing page 'https://www.notion.so/80f1c747841641e2a729fb0286390da2'
[21:49:21] CRITICAL Timeout waiting for page content to load, or no content found. Are you sure the page is set to public?
[21:49:21] INFO Parsing page 'https://www.notion.so/e861fdd6a0c247ca8bad342d2cdb05b6'
[21:50:22] CRITICAL Timeout waiting for page content to load, or no content found. Are you sure the page is set to public?
[21:50:22] INFO Parsing page 'https://www.notion.so/d5c95ef2e77349e98691b8925de7d119'
[21:51:23] CRITICAL Timeout waiting for page content to load, or no content found. Are you sure the page is set to public?
[21:51:23] INFO Finished!

Processed 1 pages in 00:04:19

If you go to a subpage directly, you'll see that it is public, but is a redirect page from Notion.

bhchiang · 2021-07-22T22:06:40Z

Thanks for pointing that out - my PR doesn't handle subpages. When parsing the subpages (sub_page_href), the www.notion.so part should be replaced with {site_name}.notion.site.

I tried a quick fix but there are a few edge cases in the code that I am probably missing, so not submitting a PR yet.

EveraertJan · 2021-08-11T09:16:16Z

Hi,

I,m currently running into the same issues. Is there any fix available?

specbug · 2021-08-16T13:44:19Z

Thanks for pointing that out - my PR doesn't handle subpages. When parsing the subpages (sub_page_href), the www.notion.so part should be replaced with {site_name}.notion.site.

I tried a quick fix but there are a few edge cases in the code that I am probably missing, so not submitting a PR yet.

@bryanhpchiang can you post the partial fix here? Others can try it out and help in fixing the edge cases.

joshkmartinez · 2021-08-17T21:47:09Z

I'm having this issue as well. Would appreciate the partial fix if possible @bryanhpchiang

PiktCai · 2021-08-19T08:49:33Z

I am not a developer, but there is a quick way to make it work properly. Actually, by simply editing the links at lines 582-584 of loconotion/notionparser.py , it works.
before editing:

            if sub_page_href.startswith("/"):
                sub_page_href = "https://www.notion.so" + a["href"]
            if sub_page_href.startswith("https://www.notion.so/"):
                if parse_links or not len(a.find_parents("div", class_="notion-scroller")):

after editing:

            if sub_page_href.startswith("/"):
                sub_page_href = "https://xxxx.notion.site" + a["href"]
            if sub_page_href.startswith("https://xxxx.notion.site/"):
                if parse_links or not len(a.find_parents("div", class_="notion-scroller")):

when running the program, I used python loconotion https://xxxx.notion.site/xxxx/{page-id}.
Hope this would help.

sunz1e · 2021-08-23T02:32:03Z

Created a PR to Use custom new Notion url format https://xxxx.notion.site instead of default one
Saw an issue where subfolder is expected in case of link of format https://xxxx.notion.site/xxxx (faced during parsing my website). Fixed that as well.

sunz1e · 2021-08-25T13:35:59Z

@bryanhpchiang could you please pull the PR and verify if its working for you as well?

bhchiang · 2021-08-25T19:53:49Z

@meSunnySrivastava

Thanks for putting together this PR. Confirming that it did work for my website to parse subpages.

The only issue is that bullet points are now missing.

EDIT:
I see that this was supposed to be fixed by #73, and that your PR merged those changes as well.

EDIT:
Deleting my dist/ + regenerating fixed the issue. The PR looks good to me, thanks!

sunz1e · 2021-08-26T15:57:33Z

Sorry I had to close the old PR because I pushed to my master directly. :)

leoncvlt · 2021-09-07T07:16:20Z

PR has been merged, thanks all!

jamesdeluk · 2021-09-14T05:46:44Z

I'm still getting the timeout issue. Exact same as the original post above.

The page is set to public:

The link is https://jamesdeluk.notion.site/James-IT-Notes-9969909992c04b5ba3a734cdf0a74530

(The Copy Link button gives https://www.notion.so/jamesdeluk/James-IT-Notes-9969909992c04b5ba3a734cdf0a74530, which forwards to the above).

jamesdeluk · 2021-09-22T05:36:56Z

Thought I'd try this again with the new Notion update. A couple things:

Trying to access the .site page itself fails:

And webdrive.log loops this:

[1632288655.436][INFO]: Waiting for pending navigations...
[1632288655.437][INFO]: Done waiting for pending navigations. Status: ok
[1632288655.445][INFO]: Waiting for pending navigations...
[1632288655.447][INFO]: Done waiting for pending navigations. Status: ok
[1632288655.447][INFO]: [edc259a3fc220da0c2d6ba0789803d04] RESPONSE FindElements [  ]
[1632288655.957][INFO]: [edc259a3fc220da0c2d6ba0789803d04] COMMAND FindElements {
   "using": "css selector",
   "value": ".notion-presence-container"
}

leoncvlt · 2021-09-30T07:26:54Z

Well, that's not gonna work regardless because you're not logged in, so the script is unable to find the notion-presence-container div which is present on every notion page - it's gonna work with public pages only.

jamesdeluk · 2021-09-30T08:47:22Z

That's my confusion though. I am logged in, and the page is public.

leoncvlt · 2022-03-06T09:02:59Z

Just checking @leshchenko1979, is this fixed by #92?

2m · 2022-03-23T06:27:04Z

I am using current master version of loconotion with the new style URLs and it seems to work fine: https://github.com/2m/nemunasring/blob/main/nemunasring.toml#L2

sueszli · 2023-03-27T22:29:56Z

Since Notion updated all URLs for hosted pages (see: #134) this ticket is no longer an enhancement, but a permanent bug.

We resolved it in our fork here: https://github.com/sueszli/notionSnapshot/

leoncvlt added the enhancement New feature or request label Aug 4, 2021

sunz1e mentioned this issue Aug 23, 2021

Use custom new Notion url format instead of hardcoded default one #74

Closed

sunz1e mentioned this issue Aug 26, 2021

Use custom new Notion url format instead of hardcoded default one #75

Merged

leoncvlt closed this as completed Sep 7, 2021

leoncvlt reopened this Sep 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for new Notion URL format #69

Support for new Notion URL format #69

bhchiang commented Jul 21, 2021

tomreitz commented Jul 22, 2021

leoncvlt commented Jul 22, 2021

tomreitz commented Jul 22, 2021

bhchiang commented Jul 22, 2021

EveraertJan commented Aug 11, 2021

specbug commented Aug 16, 2021 •

edited

joshkmartinez commented Aug 17, 2021

PiktCai commented Aug 19, 2021

sunz1e commented Aug 23, 2021

sunz1e commented Aug 25, 2021

bhchiang commented Aug 25, 2021 •

edited

sunz1e commented Aug 26, 2021

leoncvlt commented Sep 7, 2021

jamesdeluk commented Sep 14, 2021

jamesdeluk commented Sep 22, 2021

leoncvlt commented Sep 30, 2021

jamesdeluk commented Sep 30, 2021

leoncvlt commented Mar 6, 2022

2m commented Mar 23, 2022

sueszli commented Mar 27, 2023

Support for new Notion URL format #69

Support for new Notion URL format #69

Comments

bhchiang commented Jul 21, 2021

tomreitz commented Jul 22, 2021

leoncvlt commented Jul 22, 2021

tomreitz commented Jul 22, 2021

bhchiang commented Jul 22, 2021

EveraertJan commented Aug 11, 2021

specbug commented Aug 16, 2021 • edited

joshkmartinez commented Aug 17, 2021

PiktCai commented Aug 19, 2021

sunz1e commented Aug 23, 2021

sunz1e commented Aug 25, 2021

bhchiang commented Aug 25, 2021 • edited

sunz1e commented Aug 26, 2021

leoncvlt commented Sep 7, 2021

jamesdeluk commented Sep 14, 2021

jamesdeluk commented Sep 22, 2021

leoncvlt commented Sep 30, 2021

jamesdeluk commented Sep 30, 2021

leoncvlt commented Mar 6, 2022

2m commented Mar 23, 2022

sueszli commented Mar 27, 2023

specbug commented Aug 16, 2021 •

edited

bhchiang commented Aug 25, 2021 •

edited