Huge sitemap takes forever to load

[teammakdi]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

https://dev.to/sitemap-index.xml has lot of links, hence takes forever to load. Maybe we can add a timeout/url limit parameter.

Code sample

import { Sitemap } from 'crawlee';

const { urls } = await Sitemap.load('https://dev.to/sitemap-index.xml');
console.log(urls.length)

Package version

3.8.1

Node.js version

20.11.1

Operating system

Ubuntu

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[janbuchar]

Hello! I ran your code snippet and measured the execution time. The first run took ~25s, all subsequent runs took around 5s. I believe that's fine for a sitemap with 400K URLs in it.

How long did your experiment take? What kind of network connection are you using?

[teammakdi]

Hi @janbuchar, the above url was just an example, I'm concerned about urls with a huge sitemap since it can take a lot of time to return the results.

[janbuchar]

Yes, pretty much any operation with a large enough input can take a lot of time - that is a fundamental fact in programming. We are considering logging messages along the lines of "Sitemap for abc.xyz is still loading after X seconds..." for better developer experience. Do you think that would help?

[teammakdi]

Yes that would help. Do you think adding a limit parameter makes sense here? As in if the limit is 500, it'll fetch first 500 urls from sitemap and end the process.

[janbuchar]

Honestly, no. Adding another way of limiting the number of crawled pages would just complicate the API. We may consider parsing the sitemap in the background and "piping" the URLs to the crawler so that the crawling can start before the sitemap is finished loading in the future, but 1. it's fairly complicated and 2. it seems there isn't significant demand for that feature. If we decided to implement that, however, you could achieve the "stop after parsing N links from the sitemap" use case, albeit from the other side.

[teammakdi]

Cool, thanks @janbuchar.

[Sajito]

Hello, I'm actually having a problem with the sitemap loading "forever", due to a really big sitemap.
Right now I'm trying to load the sitemap from "galaxus.de", which after 10 minutes (at the time of writing) still did not finish.

I don't think there should be a limit for urls returned, but it would be really helpful if there was a way to either cancel the promise using an AbortController or define a timeout, after which the promise should reject.

[teammakdi]

@Sajito

Since this feature isn't present at the moment, wrote a small function to do the timeout part

async function executeWithTimeout (fn, args, timeout) {
  const timeoutPromise = new Promise((resolve, reject) => {
    setTimeout(() => {
      reject(new Error(`Function ${fn.name} execution timeout`))
    }, timeout)
  })
  return await Promise.race([fn(...args), timeoutPromise])
}

[Sajito]

Update here: process died after 11 minutes and 32 seconds.

[Sajito]

@teammakdi sorry, saw your reply only after posting my own response.

I tried that, problem is the request is not actually aborted, it's just ignored. That means, the code will keep running for some time until the process dies nonetheless due to the memory limit.

[teammakdi]

@Sajito

Yes, agreed.