Skip to content
/ kit Public

๐Ÿ“š Libraries for retrieving and parsing of robots.txt & sitemaps.

License

Notifications You must be signed in to change notification settings

spire-rs/kit

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 

History

15 Commits
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

Kit

Build Status Crate Coverage

Crates:

  • countio: The wrapper struct to enable byte counting for std::io::Read and std::io::Write and its asynchronous variants from futures and tokio.
  • robotxt: The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
  • sitemapo: The implementation of the Sitemap (or URL inclusion) protocol with the support of txt, xml formats and video, image, and news extensions.

About

๐Ÿ“š Libraries for retrieving and parsing of robots.txt & sitemaps.

Topics

Resources

License

Stars

Watchers

Forks

Languages