Skip to content

robotspy 0.8.0
Compare
Choose a tag to compare
@andreburgaud andreburgaud released this 05 Jul 23:55
· 40 commits to main since this release
0.8.0
daa9c72
This commit was signed with the committer’s verified signature.
andreburgaud Andre Burgaud
GPG key ID: 740FD69C978DA0AA
Learn about vigilant mode.
  • Addressed an issue raised when a robots.txt file is not UTF-8 encoded (thank you, @tumma72, for spotting the problem and providing a suggestion for a fix - #200)
  • Added a user agent to fetch the robots.txt, as some websites, such as pages hosted on Cloudflare, may return a 403 error.
  • Updated the documentation to link to RFC 9309, Robots Exclusion Protocol (REP).
  • Added a GitHub action job to execute the tests against Python versions 3.8 to 3.12.

Contributors

tumma72
Assets 2
Loading