New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FR] Exclude a list of urls #78

Open

Snowzer91 opened this issue Nov 27, 2023 · 1 comment

Snowzer91 commented Nov 27, 2023 •

edited

Loading

Can you add a feature to ignore crawling for a given list of urls?

Sample:
match: [
"https://www.builder.io/",
],
exclude: [
"https://www.builder.io/blog/",
]

Snowzer91 changed the title ~~Add feature to exclude a list of urls~~ [FR] Add feature to exclude a list of urls

Snowzer91 changed the title ~~[FR] Add feature to exclude a list of urls~~ [FR] Exclude a list of urls

miticojo commented Jan 8, 2024

In that way should works

  match: [
    "https://www.builder.io/**",
  ],
  exclude: ["**/blog/**"],

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment