You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are some UTF-8 sequences which appear the same, but are actually different code points. For example, the Icelandic character á has a nice clean UTF-8 code-point, but occasionally one will see in the wild this represented as "a" followed by a Unicode grave accent suffix.
These things will be represented as separate strings within the search engine, so maintaining a table of conversions which converts the latter into the former will help make things findable. From the user's point of view these things look identical, so it won't make sense to them that some can be found with a search, but others cannot.
A more extreme version of this is to attempt to dumb down lots of special Unicode characters to their closest ASCII representation may also help as well, since it's not uncommon for people to deliberately misspell their own language if they are stuck on an unfamiliar keyboard.
Yes, this is a can of worms.
The text was updated successfully, but these errors were encountered:
There are some UTF-8 sequences which appear the same, but are actually different code points. For example, the Icelandic character á has a nice clean UTF-8 code-point, but occasionally one will see in the wild this represented as "a" followed by a Unicode grave accent suffix.
These things will be represented as separate strings within the search engine, so maintaining a table of conversions which converts the latter into the former will help make things findable. From the user's point of view these things look identical, so it won't make sense to them that some can be found with a search, but others cannot.
A more extreme version of this is to attempt to dumb down lots of special Unicode characters to their closest ASCII representation may also help as well, since it's not uncommon for people to deliberately misspell their own language if they are stuck on an unfamiliar keyboard.
Yes, this is a can of worms.
The text was updated successfully, but these errors were encountered: