Skip to content

First version of tokenizer

Pre-release
Pre-release
Compare
Choose a tag to compare
@alesapin alesapin released this 07 Feb 16:26
· 5 commits to master since this release
v0.1
8d1a469
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Умеет разбивать текст на токены без какой-либо семантической и сантаксической информации. Версии бинарника и python-wheels под linux.

Assets 5
Loading