Replace broken decommenter script with a lexer (tandy-decomment) #55
Conversation
Previously sequential commas were not matched.
Uses tandy-tokenize and tandy-decomment from https://github.com/hackerb9/tandy-tokenize. Source code is included in adjunct directory. If the user does not have it already installed, they can run 'make tokenize'. Note, this is not the actual flex source code, but the generated C files.
Instead of requiring people to install it by hand and put it in the path, just use the copy in adjunct.
Original code: 18K Tokenized code: 16K Decommented Tokenized code: 7K Crunched Decommented Tokenized code: 6K
Steps to update to latest in the future: wget https://github.com/hackerb9/tokenize/releases/latest/download/tokenize-cfiles.tar.gz tar -zxvpf tokenize-cfiles.tar.gz mv tokenize/* adjunct/ rmdir tokenize rm tokenize-cfiles.tar.gz
|
Hi! I've fixed up the tokenizer to be much better, plus I made it so that GitHub will automatically create a new release (with appropriately bundled files) whenever the tags are updated. For example git tag -a v0.m-beta00 -m "Trial version of new release with improved tokenizer"
git push --tagsWhen I did that, it created this release in my repository: https://github.com/hackerb9/M100LE/releases/tag/v0.m-beta00 The tag name "v0.m-beta00" can be anything you want. |
The runner has used up 5min of CPU time already and is still going
trying to download and recompile zillions of packages. All I did was:
brew update
brew install gawk
Let's try it without the update first.
|
I've added some more changes which now allow all the tokenization code to check the build on the virtual MacOS environment provided by GitHub. The hope is that now you'll be able to run If it doesn't work, you can run the tokenization program remotely just by using the |
Closes #54 .
Hackerb9's m100-tokenize program is greatly improved and should (hopefully) now work on MacOS.
This new solution only relies on a working C compiler -- the actual source code (written for Lex) is not included and only the generated C files are present. The Makefile automatically compiles and uses the m100-tokenize programs in the "adjunct" directory.
Update: The only thing left to do is figure out the file format for N82 BASIC.