feat: Enhanced Web Crawler with Markdown Output and Customization

[robin-collins]

Added @mozilla/readability, jsdom, and turndown dependencies to enable markdown conversion and improve content extraction.
Updated versions for various @crawlee, @apify, and other dependencies to leverage the latest features and improvements.
Removed old versions of chalk, cli-width, figures, and other dependencies to streamline the project.
Added command-line options (-f, --outputFileFormat) and interactive prompts for output file format (JSON, markdown, human-readable markdown) and name, providing flexibility and customization.
Implemented enhanced markdown conversion with better handling of HTML elements (lists, links) for cleaner and more accurate output.
Added a "human-readable markdown" option (-f human_readable_markdown) with table of contents and "Back to Top" links for easier navigation in larger documents.
Refined URL exclusion logic in core.ts to handle query parameters containing &do= more effectively, allowing for more precise control over which pages are crawled.
Updated server (server.ts) to set content type to text/markdown for markdown output, ensuring correct rendering in browsers and other applications.

below is an example of a ./config.ts file that takes advantage of the enhancements / updates.

import { Config } from "./src/config";

export const defaultConfig: Config = {
  url: "https://docs.dopus.com/doku.php?id=scripting",
  match: "https://docs.dopus.com/doku.php?id=scripting**",
  exclude: [
    "**&do=resendpwd**",
    "**&do=register**",
    "**&do=login**",
    "**&do=logout**",
    "**&do=profile**",
    "**&do=edit**",
    "**&do=diff**",
    "**&do=revisions**",
  ],
  maxPagesToCrawl: 75,
  selector: ".page",
  outputFileName: "directory_opus.md",
  outputFileFormat: "markdown",
  maxTokens: 2000000,
};

[marcelovicentegc]

Nitpick: It occurs to me that we could store this array as a constant somewhere, so we can use it on this line and on src/config.ts.

[marcelovicentegc]

I suppose &do= is specific to some test case? Shouldn't the exclude config option take care of excluding declared patterns?

[marcelovicentegc]

Hey @robin-collins 👋 ! Thanks for opening this PR. I've left a couple comments. Simple stuff! Out of pure curiosity: what is the use case for markdown output?

[robin-collins]

the markdown output has two uses for me, a) in Anthropic Claude 'projects', and I also use the markdown version with the self-hosted libre chat that the AI has it as part of the RAG system. The human-readable format is for me to have a nice n neat markdown document for my own reference. I prefer it than hitting an online documentation site regularly.
the exclude patterns were me trying to get a nice grab of the Directory Opus documentation, perhaps a little specific but the extended exclude options make it nice n easy to keep the crawl very specific.

[frei-x]

👋 嘿 ！感谢您打开此 PR。我留下了几条评论。简单的东西！纯粹出于好奇：markdown 输出的用例是什么？

I need to get the image src