Adding relational properties to tokens?

[calculuschild]

Following the discussion on PR #2043 on these comments, I'm curious if there's any other interest in this idea to aid in traversing the token tree for people who want to extend or customize Marked. I see two issues:

1. Accessing adjacent tokens requires breaking up the translation pipeline. The walkTokens() function on its own does not have a way to reference adjacent siblings or parent tokens. An alternative is to have the user to break apart the processing into a separate Lexer step, manipulate the resulting token tree as they wish, then send the updated tokens into the Parser. This is doable but does not lend itself to a clean extension package that is easily publishable. UziTech has already suggested adding a postProcessing or preProcessing hook to handle this, which seems like a good option.

2. Tree traversal is still cumbersome. Even with access to the token tree via Lexer, determining which tokens are parents, grandparent, siblings, etc. might require a good deal of custom code for the user.

What are the thoughts on adding a few basic properties to each token as part of the Lexer process? previousSibling, nextSibling, children, and parent.

As each token is created, it would be trivial to tack on the previousSibling and parent properties since those are readily available in the Lexing loop and we already use them. We can then amend the previousSibling token to point to the current token with nextSibling. Similarly as each child token is created, we can access it's parent and append the current token to a children array.`

This would make tree traversal more intuitive and simple for extension creators. Need to check if a token is between two tokens of a particular type? if(token.previousSibling.type === 'hr' && token.nextSibling.type === 'hr') {...} Need to merge adjacent tokens together? Or check for the presence of a particular child token? Or grandparents? token.parent.parent.

[UziTech]

I'm having a hard time coming up with a valid use case for this where breaking apart the lexer and parser step wouldn't already solve.

My biggest objection to this is that it would slow down marked for most users (even if it is just a little bit) with no benefit. And it wouldn't benefit the people who would use it a lot more than they can already do it.

If there is a scenario where something useful couldn't be done without this than the benefit might out weigh the cost of slowing down everyone else.

For instance I think #2043 is beneficial because it enables things that aren't possible without rewriting the lexer and parser (arguably the only reason to use marked) and for anyone who isn't using any extensions it is just a few if statements in reduced speed.

[calculuschild]

How about this: If I build it and it turns out to have negligible impact on speed, do you have another reason to not do it?

[UziTech]

No, if there was an implementation that doesn't hurt the benchmarks and adds this functionality I would be fine with it.

[calculuschild]

Ok. I will take a crack at it then.

[UziTech]

it prevents packaging into a clean, distributable extension.

Adding a hook that gives the output of the lexer would solve this and not add any reduced speed.

[calculuschild]

Right, that solves 1) above. There's still a benefit I think to having built-in direct access to siblings and parents rather than needing to build that logic up from scratch strictly from an "ease-of-use" standpoint, 2) above. Unless we add that in as part of the hook; it processes the tree and adds the relational properties before spitting it out to the user.