Trim UA block list for false positives #189
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 15 commits
February 5, 2023 23:30
…in the popular Redmi Note series of phones and tablets, made by Xiaomi: <https://en.wikipedia.org/wiki/Redmi#Redmi_Note_Series>
…othing inherently malicious
…eApp" in its UA string
…icrosoft Teams. Their agent uses UA strings like this: MicrosoftNinja/1.0+Teams/1.0+(ExchangeServicesClient/0.0.0.0)+SkypeSpaces/1.0a$*+ This breaks MS Teams integration with MS Exchange, possibly more.
…e of robot) "UptimeRobot", which is supposed to be limited - not blocked
… block legitimate WebDAV clients correctly identifying as such. WebDAV is a very widely used official HTTP standard, and is the foundation of popular solutions like Nextcloud and ownCloud.
…ent stable release), and the version codename string is used as part of UA for several of its packages, such as Python's pip
…, in terms of features such as accessibility. If you're receiving requests from them, it's likely because you asked (and paid) them to.
…to archive digital content and create a library for current and future generations. They are good guys, and if one person is annoyed by their work not strictly adhering to robots.txt, let that person block it locally, rather than cause this level of collateral damage for a good cause. Closes mitchellkrogza#87
…ighting cybercrime on a plethora of different fronts. When they do mine technical data, it's for a good cause, that benefits all of us and makes the Internet a safer place. Their good efforts should not be blocked.
…rability scanning. It doesn't do harm, it merely discovers weakness in your setup, which makes it possible for you to fix them. Hiding from it doesn't make you more secure - on the contrary.
…e spiders, and embedded views in apps. Closes mitchellkrogza#133
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After 1,5 months of using your UA block list in a production environment of a certain size, we had to make some local alterations to weed out false positives we were encountering. I'm submitting them all here, together with an explanation and justification for each, presented as individual commits in case you feel they need to be cherry-picked.
Do note that we're not using the list "as intended", but rather consume the source list directly to use in HAProxy, where we do case-insensitive sub-string matching against the client's UA. If you feel there's a better (unified) way to make use of this data - such as use word boundaries or case sensitivity or whatnot - I'm all ears.
Thanks for all your hard work!