Code

URL doctor PoC#900

Code

Open

mogery wants to merge 20 commits intomainmendableai/firecrawl:mainfrom mog/doctormendableai/firecrawl:mog/doctorCopy head branch name to clipboard

+471-149

Commits

Commits on Nov 14, 2024

• doctor first iteration

  mogery

  committed
  faf11ac

• feat: turn into API

  mogery

  committed
  9298a05

Commits on Nov 15, 2024

• fix(scrapeURL): adjust error message for clarity

  mogery

  committed
  34dfc3c

• Create README.md

  

  ericciarla

  authored and

  mogery

  committed
  616841c

• fix(sitemap): scrape with tlsclient

  mogery

  committed
  590824f

• feat(v1/batch/scrape): webhooks

  mogery

  committed
  a4bcd54

• Update package.json

  

  nickscamara

  authored and

  mogery

  committed
  ff31725

• fix(batch/scrape/webhook): add batch_scrape.started

  mogery

  committed
  15199ce

• Nick: sitemap only

  

  nickscamara

  authored and

  mogery

  committed
  68f23e7

• Nick: moved away from axios

  

  nickscamara

  authored and

  mogery

  committed
  1a07a29

• Update fireEngine.ts

  

  nickscamara

  authored and

  mogery

  committed
  daefe00

• Nick:

  

  nickscamara

  authored and

  mogery

  committed
  cb02d77

• Update map.ts

  

  nickscamara

  authored and

  mogery

  committed
  115fad9

• fix(scrapeURL/playwright): JSON body fix

  mogery

  committed
  008ec88

• fix(v1/batch/scrape): raise default timeout

  mogery

  committed
  95b57b4

• fix(v1/types): better timeout primitives

  mogery

  committed
  c2be6cc

• fix(log_job): add force option to retry on supabase failure

  mogery

  committed
  cc19cc4

• fix(crawler): treat XML files as sitemaps (temporarily)

  mogery

  committed
  43642bd

• fix(scrapeURL/fire-engine): timeout-less scrape support (initial)

  mogery

  committed
  af816c7

• fix(scrapeURL/fire-engine): wait longer if timeout is not specified

  mogery

  committed
  519f4ff