WIP: Adding "push lexer" functionality. #476
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
This is my very first (beta) proposal for adding "push lexer" functionality to flex.
The rationale is since bison has a push mode, adding this functionality allows the whole lexer/parser to yield instead of doing a blocking wait for more input data. This is useful for example in embedded systems, where micro controllers do not have threads, but could benefit for online protocol parsing.
How it works:
some (most) local variables from YY_DECL have been transferred to the yy_guts structure.
YY_DECL can now return YY_STALLED if no data is available to complete the current lexeme.
This happens if yy_fill_buffer is equal to 2 (maybe this should be changed to a #define) and a YY_STALLED (preprocessor) symbol exists.
There is a new yy_append_bytes function that appends bytes to the current input buffer (it can increase its size, but does not replace it like yy_scan_bytes).
Status:
All tests pass. The tests will be adapted so they run also in the push lexer configuration, where that makes sense.
Please give me some feedback on the design decisions I made, and what should be changed (for example usage of a % keyword).
Cheers,
Jérôme.