-
-
Notifications
You must be signed in to change notification settings - Fork 186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stable v2 release (API changes) #108
Comments
Alright, feature request time
|
@ceymard Perhaps done better in participle, but currently we use |
@hinshun I'm doing doing something similar at the moment ; I just wish for something to get a match easily, without having to resort to that kind of trick. |
This speeds up parsing by 5-10%: benchmark old ns/op new ns/op delta BenchmarkEBNFParser-12 143589 129605 -9.74% BenchmarkParser-12 395397 375403 -5.06% BenchmarkParticipleThrift-12 202280 191766 -5.20% BenchmarkParser-12 7724639 7114586 -7.90% See #108.
This includes tokens elided by Elide(), but not tokens elided by the Lexer. See #108.
This functionality is now included natively. Any node with a field |
You can also now capture directly into a field of type |
Do the |
Yes they do. |
Do they include the elided ones as well ? |
Yep! |
This speeds up parsing by 5-10%: benchmark old ns/op new ns/op delta BenchmarkEBNFParser-12 143589 129605 -9.74% BenchmarkParser-12 395397 375403 -5.06% BenchmarkParticipleThrift-12 202280 191766 -5.20% BenchmarkParser-12 7724639 7114586 -7.90% See #108.
This includes tokens elided by Elide(), but not tokens elided by the Lexer. See #108.
This speeds up parsing by 5-10%: benchmark old ns/op new ns/op delta BenchmarkEBNFParser-12 143589 129605 -9.74% BenchmarkParser-12 395397 375403 -5.06% BenchmarkParticipleThrift-12 202280 191766 -5.20% BenchmarkParser-12 7724639 7114586 -7.90% See #108.
This includes tokens elided by Elide(), but not tokens elided by the Lexer. See #108.
Now that Participle has proven its initial concept, I think it's time to clean up the API. This will be a backwards incompatible change.
Work has started in the v1 branch.
Pos
/EndPos
to support capturing the full range of tokens the node matched, includingElide()
ed tokens. (2ace05e)Mapper
to eliminate the need forDropToken
. (f82f615)lexer.Token
and[]lexer.Token
. (3b1f151)Maybe:
participle.Elide()
support so that elided tokens can be captured explicitly by name (but also see next point).Support streaming tokens from anio.Reader
- currently the full input text is read.Refactor PeekingLexer so it doesn't consume all tokens up front.Once the API is stable, some additional changes would be welcome:
Regarding streaming, I'm not convinced this is a worth the considerable extra complexity it will add to the implementation. For comparison, pigeon also does not support streaming.
Additionally, to support the ability to capture raw tokens into the AST, participle will need to potentially buffer all tokens anyway, effectively eliminating the usefulness of streaming. It also vastly increases the complexity of the lexers, requiring three paths (
io.Reader
,string
and[]byte
),PeekingLexer
, etc.This increased complexity is mainly due to the lookahead branching, and the lexer needs to have a similar implementation to the rewinder RuneReader code (https://play.golang.org/p/uZQySClYrxR). This is because for each branch the state of the lexer has to be stored but also, additionally, as each branch progresses it needs to preserve any new tokens that are buffered so that if the branch is not accepted the parent can remain consistent.
There's also a non-trivial amount of overhead introduced for reading each token, as opposed to the current PeekingLexer which is just an array index.
The text was updated successfully, but these errors were encountered: