Refactor Tokenizer::tokenize by not relying on Tokenizer::char_type a…#392

Open

baxter2 wants to merge 1 commit intomojombo:masterfrom baxter2:refactor/Tokenizer.tokenize

+5-11

Commits on Jun 7, 2019

Refactor Tokenizer::tokenize by not relying on Tokenizer::char_type and shortening methods length.
baxter2
committed