List of Links that aren't working

[calebpeffer]

Hi all, Caleb from @mendableai here.

We will start collecting links that have issues here so we can fix them. If you'd like to add a link to the list, please follow this format.

Link: https://news.ycombinator.com/
Endpoint: /scrape, /crawl
Issue: Tables don't seem to be parsed correctly. The tables return as HTML instead of as a formatted table.

We will fix them as quickly as possible.

[calebpeffer]

Link: https://privacy.patreon.com/policies
Endpoint: /scrape /crawl
Issue: fails all fallbacks

[rafaelsideguide]

I tested the system once and it failed, but it worked when I tried again. Non-deterministic behavior may be related to the fallback conditions?

[nickscamara]

Yep. It seems to be a bit better now. Tried a few times and even triggering most fallbacks, it was still able to scrape the content.

[rafaelsideguide]

Works with /scrape. "No pages found" for /crawl though

[calebpeffer]

Link: https://scholarship.law.uci.edu/cgi/viewcontent.cgi?article=1584&context=ucilr
Endpoint: /scrape /crawl
Issue: All content is Null

[nickscamara]

#29 fixes this.

[calebpeffer]

Link: https://docs.streamlit.io/
Endpoint: /crawl
Issue:

[nickscamara]

@calebpeffer hmm interesting. I believe we could add a dynamic routes and dynamic paths like [route]/path/path2/[dnyamic].cache in the excludes and includes to be able to filter out unwanted paths/files.

[igalmarino]

Link: https://support.greenpay.me/portal/es/kb/support
Endpoint: /crawl
Issue: It only gets the links on the header, but not the rest on the body of the page.

[igalmarino]

i do not know if im doing this right.

Do i need a new version of firecrawl-py to test these fixes.

https://pypi.org/project/firecrawl-py/

Or is there any other way to test it?

Tnanks

[nickscamara]

No need for a new version, we pushed a server change so it should just work with what you currently had.

[igalmarino]

I have been testing it and still only get 7 results (there are mora than 40 links on the page)

[nickscamara]

@igalmarino Pushing a change that might help with this... Will be up in 1 hour. Can you let me know if it works? Thank you!

[igalmarino]

It is working now. Thank you !

[calebpeffer]

Link: https://apps.dos.ny.gov/publicinquiry/
Endpoint: /crawl
Issue: this site displays with a filter and javascript, this is a nasty one.

Note: some other websites that may or may not work in the same space:

1. Alabama - https://www.sos.alabama.gov/
2. Alaska - https://www.commerce.alaska.gov/web/cbpl/Corporations.aspx
3. Arizona - https://azsos.gov/business
4. Arkansas - https://www.sos.arkansas.gov/business-commercial-services-bcs
5. California - https://www.sos.ca.gov/business-programs/business-entities
6. Colorado - https://www.sos.state.co.us/

Note: Email [email protected] once complete

[nickscamara]

Yea, first url is really weird...

[calebpeffer]

Links: https://investors.snapone.com/static-files/c0d11e9f-f69b-48e2-b9a1-3cdb4992ed38
https://investors.snapone.com/static-files/365445cc-1182-4707-ab07-552c8c0a0bdf
https://investor.shutterstock.com/static-files/f5b52aee-0070-4922-acb6-9886feee789d
https://investor.shutterstock.com/static-files/79474632-2f5f-46a7-b481-e5cbd5274915

Endpoint: /scrape

returning
%PDF-1.7 %���� 1 0 obj <>/Metadata 115 0 R/ViewerPreferences 116 0 R>> endobj 2 0 obj <> endobj 3 0 obj <>/ExtGState<>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 612 792] /Contents 4 0 R/Group<>/Tabs/S/StructParents 0>> endobj 4 0 obj <> stream x��][S��w��C=v�0EI%�T��a����abf�C���g��ж������h�u�]U*]2S�_�R����z�iq�n��v�������k����_�g?o�����/������鷏k���rq����o_5__������6]#��0��n�������������'�!��Xs���
id rather it fail gracefully

Extra context:

[nickscamara]

#29 fixes this.

[calebpeffer]

Link: https://www.rareskills.io/zk-book
Endpoint: /scrape, /crawl
receive a 502, could you please help? from Dmitry Ukhanov on discord

[nickscamara]

Fixed ✅

[nickscamara]

DM'd him!

[calebpeffer]

Link: https://developers.notion.com/reference/post-page
endpoint: /scrape /crawl

[nickscamara]

Fixed ✅ @calebpeffer can you let them know? and see if it works better now for them?

[garrettfrohman]

Link: https://docs2.hubitat.com/
Endpoint: /crawl
Error: No pages found

[nickscamara]

@garrettfrohman Fixed ✅

[skye0402]

Link: https://api.sap.com/products/SAPS4HANACloudPrivateEdition/apis/all
Endpoint: both fail /scrape, /crawl (but want /crawl)
Issue: No reply at all or just a response that shows the non-dynamic content

[ericciarla]

Link: www.wakestreamventures.com
Endpoint: /scrape
Issue: No page found

Link: www.westlygroup.com
Endpoint: /scrape
Issue: No page found

Link: www.waldenvc.com
Endpoint: /scrape
Issue: No page found

Link: www.villageventures.com
Endpoint: /scrape
Issue: No page found

Link: www.zoiccapital.com
Endpoint: /scrape
Issue: No page found

[ericciarla]

Notify neuralweb.tech when done

[nickscamara]

https://www.citizensadvice.org.uk/scotland
Endpoint: /scrape
Issue: 1 page found

[rafaelsideguide]

https://support.carta.com/s/article/master-subscription-agreement
Endpoint: /scrape
Issue: JSON response content doesn't contain the page content

[olaf-lexemo]

Link: https://www.cmegroup.com/markets/agriculture/grains/corn.quotes.html#venue=globex
Endpoint: /scrape
Issue: using python sdk i get a timeout. using the playground i get 'No pages found'

[calebpeffer]

Link: wise.com
/crawl
i tried wise.com a few times. with 500 limit and depth = 10 and fast crawl. The crawl hasn't completed for any of those runs. although each time firecrawl did not return any additional pages after 146.

Notify Blee via slack once fixed

[theGoillot]

link: https://help.innovorder.fr/
/crawl

We don't get the depth, only the first URL and not the content behind

[calebpeffer]

link: https://www.parcnl.ca/documents/nalvma_sept2014.pdf, https://www.parcnl.ca/documents/nalvma_sept2014.pdf
/scrape

PDF parse failure

@bardia-pourvakil

[calebpeffer]

link: https://www.solvhealth.com/privacy, https://support.carta.com/s/article/master-subscription-agreement
Unknown Endpoint
Unknown Error

@bardia-pourvakil

[nickscamara]

Link: https://docs.jup.ag/guides
Endpoint: /crawl
Returning only 1 page

[gratitude5dee]

Link : https://www.hyatt.com/hyatt-regency/en-US/sfors-hyatt-regency-san-francisco/rooms
Endpoint : /crawl
Return Null

[chrisjainsley]

Link: https://eliasstravik.com/
Endpoint: /crawl
Issue: Returns an empty array (nodejs sdk)

[cavc02]

Link: wearebond.myshopify.com (and other XXXX.myshopify.com domains)
Endpoint: /crawl
Issue: Returns only main page but no subpages.

[calebpeffer]

Link: https://electionresults.utah.gov/results/public/ironcountyutah/elections/2023-Nov-General
endpoint: /Scrape
Issue: unable to grab dynamic content and results of the election.

[thatsjonsense]

Link: https://pictographic.io/
endpoint: /scrape
Issue: no content found