Links are now extracted after applying excludeTags

[txrp0x9]

fixes #701

[nickscamara]

I think the problem with this is that we rely on this function for our /crawl, which will end up failling to grab all the links if we don't pass the raw version

[nickscamara]

@mogery lmk if im wrong

[txrp0x9]

A global code search did not reveal any usage of the linksOnPage field anywhere other than api return
I believe the crawler uses a separate extractLinksFromHTML function

firecrawl/apps/api/src/services/queue-worker.ts

Line 374 in 8a4f4cb

crawler.extractLinksFromHTML(rawHtml ?? "", sc.originUrl),

and

firecrawl/apps/api/src/scraper/WebScraper/crawler.ts

Line 382 in 8a4f4cb

links.push(...this.extractLinksFromHTML(content, url).map(url => ({ url, html: content, pageStatusCode, pageError })));

defined as

firecrawl/apps/api/src/scraper/WebScraper/crawler.ts

Lines 322 to 337 in 8a4f4cb

	public extractLinksFromHTML(html: string, url: string) {
	let links: string[] = [];
	const $ = load(html);
	$("a").each((_, element) => {
	const href = $(element).attr("href");
	if (href) {
	const u = this.filterURL(href, url);
	if (u !== null) {
	links.push(u);
	}
	}
	});
	return links;
	}

[nickscamara]

Right makes sense.