You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It is difficult to segment a text into tokens in some languages(such as Chinese), there are many hard works need to do to implement better tokenizer. For this reason, sometimes the tokenizer is implemented in other programming language, even in other services( in microservices architecture). In this case, support async version of tokenizer to request tokens between services is required.
It is difficult to segment a text into tokens in some languages(such as Chinese), there are many hard works need to do to implement better tokenizer. For this reason, sometimes the tokenizer is implemented in other programming language, even in other services( in microservices architecture). In this case, support async version of tokenizer to request tokens between services is required.
PR: #21
The text was updated successfully, but these errors were encountered: