Feature: Variables in Brews

[G-Ambatte]

Capturing conversation from Gitter (edited for clarity and length):

G.Ambatte: I posted my Adventure Template in the subReddit earlier today, and @Gazook89 made a comment about my use of variables as stand ins for text substitution. This got me thinking that - hypothetically, at least - a list could be added to a brew's metadata consisting of pairs of strings - the first being a key and the second a value, used for text substitution. An arbitrary marker could then be used to identify the keys - say, $$key$$ - which is then somehow handled as value instead.
This line of thought led me to the \client\homebrew\editor\editor.jsx, where the render function calls the <BrewRenderer [...]> with one of the passed values being this.props.brew.text.
So my thinking is that if this.props.brew.text were passed through some sort of replaceAll function that could process said key-value pairs from the brew's metadata, then it would be passed into the BrewRenderer and appear in the output window.
And, if I understand it correctly, this point even precedes Markdown parsing, so one might be to create a key like $$pagebreak$$ with a value of \page/n<div class="pageNumber auto"></div> and then not have to enter the same page set up on every single page.

Eric Scheid: There have been similar thoughts with various MD tools. If we were to do this, we should put the UI for defining the vars into a new editor pane (i.e. not some weird markdown text to be not-rendered). Just like we’re doing with CSS.
The variable markup I’ve seen about are {{variable}} and {$variable}.
The former conflicts with our v3 {{blocks}} markup.

G.Ambatte: Yeah, I believe that's one of the things planned for v3 - tabs for CSS and Metadata. Or at least, it was at one point. Embedding a \page and <div class='pageNumber auto'> looks to work the way I thought it might, so this type of system could also be used to create shortcuts for repetitive hunks of code.

Trevor Buckner: Yes this is definitely something we could do if we agree on a good syntax.
Here's another approach that builds off of the existing 'reference Link' syntax. https://www.brianchildress.co/variables-in-markdown/
By the way, I've been adding significant changes to the Marked.js library that should allow easier/more stable creation of our custom Markdown extensions like this.

Gazook89: That article is frustratingly brief, at least for me. How does it work with just substitution rather than creating a link ? JS?

G.Ambatte: I agree - when he said "example", I was looking for an implementation, not just the bare bones of an idea.

Trevor Buckner: We can easily introduce a signifier character to distinguish between links and direct variable text. Maybe put it in quotes, or use two colons: [myvar]::hello guys
My initial thought is having a user type out [myVar]:: Ragnar the Red, which they could find in a pre-made snippet, is simpler than asking them to dig through a menu. And on our end, we end up doing some parsing either way we approach it, but hiding the output is trivial.
Marked.js makes it pretty easy to re-parse your "snippets" at any point. We make the "reflink" parser change [MYVAR] to **myBoldSnippet**, and then immediately feed that substring back into Marked again so the result is <strong>myBoldSnippet</strong> before it ever gets sent to the browser.
We already do that with the mustache divs.

---

At this point, the topic changed in the Gitter conversation.

In summation: while the definition syntax of [key]:: value appears to have been agreed upon, the usage syntax is still yet to be clearly defined.

[ericscheid]

IIRC, an editor pane for variables was suggested at one point.

While a worthy long term goal, the initial implementation would be via syntax within the brew.

I would also suggest that maybe we could establish a convention of code-fencing the definitions, something like this.
To better support once we have variables carved off into a separate editor.

    ```variables
    [key1]:: value
    [key2]:: value
    ```

[ericscheid]

Since this was written, text::more text is now parsed as meaning term::definition to render as <dt>term</dt><dd>definition</dd>

An alternative syntax would be required.

[ericscheid]

Since the variable declarations are being code fenced, we could simply use YAML to declare the keys:values (which is less crufty than using e.g. JSON)

```variables
name: fred
abilities:
  STR: 18
  INT: 12
  WIS: 19   # via treant heritage feature
background:
  born to a noble
  abandoned to pixies
  raised by a treant
```

(with maybe some restrictions on variable names to suit using those variable names elsewhere)

[ericscheid]

These code fenced variable blocks could appear anywhere within the brew code (same as <style /> blocks can), and a separate editor pane could also be made for variables (like we did for styles)

[ericscheid]

Collating various ideas seen in the wild:

• simple math: e.g. define var1 as 17, define var2 as floor(var1/2)-5 (well, not exactly "simple", but also not weirdly complex)
• template strings a la madlibs: Define template-name to be "You found a _____ _____ a _____." and then use as ⇒ {template-name(NOUN,VERB,ANIMAL)} ⇒ "You found a NOUN VERBING a ANIMAL."
• auto-incrementing a variable on use (e.g. ##### TABLE {$next(table-nums)}: FOOBAR .. well, that's just an extension of simple math with a bit of persistence
• define a variable as the current page number - e.g. write [obscureref]::page() into a page (or whatver the syntax will be), and then elsewhere the author can write "See blahblah on page {obscureref}." That is, a dead-tree version of hyperlinks.
• apply variable substitution before markdown rendering, so md syntax can be included in variable definitions

Some are going well beyond the simplest implementation of course.

[ericscheid]

Yes, the syntax is the [ ] { } (etc) stuff — the exact special sequence of text and punctuation to specify the intention.

There are two bits of syntax we haven't figured out for with variables. First is how to declare a variable name and it's value or formula, and the second is what code is needed to cause that value to be literally expressed into the rendered result. Possibly some way that both could be accomplished in the one string of code too.

We've come up with a couple of ways in previous comments, only to find that other features have in the meanwhile been developed and used those special characters. (Looking at you {curly-injection}).

This is all theorising at this point, mostly exploring the possibilities to see if the effort would be worth it.

---

BTW, I wouldn't expect there would be anything to be done in the style editor with regard to variables. (CSS has it's own variable syntax already).

[ericscheid]

An example syntax might be like this (from gitter):

The variable definition syntax is [key]<<[value] on a line by itself
Variable calling is [key]>>

[ericscheid]

Making a note for deeper thought later: perchance.org does some interesting thing with variables and expressing and evaluating.

Similar: https://orteil.dashnet.org/randomgen/?do=create

[Gemanelo]

Thanks for helping, but nothing is working in homebrewery. The << aren't working at all, and the combinations I try are useless.
Homebrewery is based on Markdown, and I found some articels which said that this define and calling is not implemented/supported. I hoped the CSS part could do the trick, but it doesn't.

So I don't think it's possible to do it. But thank you anyway!

[calculuschild]

@Gemanelo None of these options actually exist yet. We are just using this page to discuss some possible options for the future.

[ericscheid]

More discussion here, including solving the Halting Problem.

[ericscheid]

We could also look to how Twine handles variables

https://twinery.org/cookbook/terms/terms_variables.html
https://klembot.github.io/chapbook/guide/state/the-vars-section.html

[ericscheid]

Two ideas from reddit ..

1. variables could be scoped to an enclosing block
2. variables could be formed into objects

An example might help:

Consider a snippet template for a monster, NPC, magic item, or even spell. Within that block various variables are declared (e.g. character abilities of strength/wisdom/et al), and are available to be used within that block, either as simple outputs or in computed expresssions (e.g. a skill chance that is affected by an ability score).

That much so far doesn't really call much need for scoping or objects.

However, within that same block one could declare another variable — one which is not scoped — and stuff all those local variables into that object.

Like this:

{{npc,framed,wide,fancy
[_name] := Maggie the Magnificent
[_ability.INT] := 17
[_ability.WIS] := 11
[_ability.STR] := 9
[protagonist.name] := [_name]

<!-- stat block formatting shenanigans -->

Behold, [_name] has an intelligence of [_ability.INT]!

}}

Later, much later ...

Our protagonist, [protagonist.name], has been presented with a dilemma that even their [protagonist.stats.INT] intelligence will struggle with.

Above, the underscore _ preceding the variable name )e.g. [_name] signals the variable is scoped to the enclosing block. The use of dots . is a syntax marker of object structure (e.g. [_protagonist.stats.INT]), and thus by combining both features it is possible to a) use a standardised snippet template without having to rename all the variables, but b) be able to refer to values specific to one block outside of that block by assigning those locally scoped variables to a non-scoped variable.

[ericscheid]

It occurred to me that it might even be better to try this

{{npc,framed,fancy
[_.name]:= Maggie
[_.stats.STR]:= 10
[_.stats.INT]:= 17
[_.stats.WIS]:= 12
[_.hitpoints]:= 22

... blah blah template formatting shenanigans ...

[hero]:= _
}}

{{npc,framed,fancy
[_.name]:= Red Baron
...
[bbeg]:= _
}}

Say hello to my little friend, [hero.name]!  
They are here to defeat you, [bbeg.name]!

That is, reserve a solitary underscore _ as a special case variable name meaning "this is an object variable that is block scoped". That way all the locally scoped variables would be declared as object items of _, and all the locally scoped variables could then be assigned in one line [globalVarName]: = _

Then again, there can come to be cases where a locally scoped variable is needed for [reasons], but shouldn't be exported. These could be declared by prefixing the varname with an underscore, like this: _locallyScopedVarName. The solitary _ is then just a special case convention of a locally scoped varname.

[ericscheid]

Similarly, it should be possible to load up the block scope variable _ from a regular variable, like this:

{{someblock
[_]:= [varname]
...
}}

That way all the block scope variable values might get defined in some other space and not clutter up the brew code. (e.g. a code block or even a separate editor pane just for variables)

{{monster,framed
[_]:= [monsters.froghemoth]

### [monster.name]
...all the rest of the monster block template
}}

[ericscheid]

Some of our template snippets have smarts backstage to generate interesting content (instead of the exact same static text every time).

Elsewhere, there has been discussion (#1870) of being able to dynamically edit/regen these template snippets (vs. needing to manually edit the raw output).

I'm thinking these two features could be leveraged, perhaps like this:

{{monster,frame
[_.name]:= Pinstriped Famine Sheep
[_.type]:= Large guy
[_.alignment]:= wordy evil
[_.AC]:= 14 (chain mail, shield)
[_.HP]:= 7(1d4+5)
[_.speed]:= 36ft.

## [_.name]
*[_.type], [_alignment]*
___
**Armor Class** :: [_.AC]
**Hit Points**  :: [_.HP]
**Speed**       :: [_.speed]

etc
}}

Here locally scoped variables are declared for each property or attribute of the block (and then used in the block) .. and the snippet dynamic editor widget just looks for locally scoped variables to update with new values (either manually entered via a widget-wizard UI, or dynamically generated by scripting)

This way the {{block }} can be edited and messed with (e.g. "Armor Class" translated to "Classe d'armure"), and the widget-wizard is still able to find the bits it needs to change.

---

Side note: not 100% happy with this, as it means a snippet template doubles in size with all the variable declarations.

[ericscheid]

From gitter:

Trevor Buckner @calculuschild 05:40
Another thing I thought of, is we could copy the syntax from GitHub for emojis, where you have a :smile: which will turn into 😊, but use it for variables instead. And then to define it something like :myVar: my text

Eric Scheid @ericscheid_twitter 12:55
I like :var: for the simplicity. I think it needs a tiny bit more for defining though..
Here is :protag:, he will save us! -- is that expressing :protag:, or defining it's value?
:protag: enters the room to great fanfare -- even requiring it start a line is ambiguous.

OK, if we use :varname:=value for defining (and requiring that such be on whole lines), and then :varname: for expressing, we can use this regex for finding variable declarations (and incorporating the ideas of _ indicating a local-scoped variable, and . for object properties):

/^:((?:_(?:\.[a-z0-9]+[a-z0-9_]*)+)|(?:_?[a-z0-9]+[a-z0-9_]*(?:\.[a-z0-9]+[a-z0-9_]*)*)):[ \t]*=[ \t]*(.*)$/gim

See https://regex101.com/r/ViICSZ/ for details and test cases.

[ericscheid]

In code, it might be better to construct the regex, instead of splatting the literal regex.

Something like this..

const varname = "[a-z0-9]+[a-z0-9_]*";
let regex = new RegExp ( `^:((?:_(?:\\.${varname})+)|(?:_?${varname}(?:\\.${varname})*)):[ \\t]*=[ \\t]*(.*)$`, 'gim');

[ericscheid]

Starting a line with : could clash with a proposal for a modified syntax for dl dd.

term
: definition 1
: definition 2

See #2340

[ericscheid]

NB. We now use a double-colon for signalling dt/dd.

term :: definition

And soon (maybe)..

term :: defn 1 :: defn 2 :: defn 3

term :: defn 1
:: defn 2
:: defn 2

term :: 
:: defn 1
:: defn 2
:: defn 2

term :: defn 1a :: defn 1b
:: defn 2
:: defn 2

[ericscheid]

If we have structured variables like [monster.stats.STR], then it could be useful to have a variable definition that loads the variable value from an API.

```variables
monsters.froghemoth: @http://www.api.com/monsters?FROG345
monsters.xathalex: @https://www.dnd5eapi.co/api/monsters/adult-black-dragon/ 
monsters.xathalex.name: Xathalex the Bold
...etc
```

A couple of such sources:

• https://api.open5e.com/monsters/adult-black-dragon/
• https://www.dnd5eapi.co/api/monsters/adult-black-dragon

[ericscheid]

Which then lets people start with a variable based snippet, and simply modify one line..

{monster,frame
```variables
[_]=@https://api.open5e.com/monsters/adult-black-dragon/
```
## [_.name]
*[_.type], [_alignment]*
___
**Armor Class** :: [_.AC]
**Hit Points**  :: [_.HP]
**Speed**       :: [_.speed]

etc
}}

[ericscheid]

We can flag code blocks for specific purposes by exploiting the code-language syntax .. but there's no built in way to signify specialness for inline code spans.

A possibility is to treat code-spans that start with a square-bracket [ as meaning "here is code that is to be interpreted using our variables extension" (i.e. the value of varname is inserted). That, and [varname]=<expression> as assigning a value to $varname.

Thus, this:

<!-- assign a value to a variable -→
`[hero.name]=Hrothgar the Mighty`

<!-- use that variable -→
On a sunny day, we meet our hero `[hero.name]`. [ed: does it have to be sunny?]

Renders as:

<p>On a sunny day, we meet our hero Hrothgar the Mighty. [ed: does it have to be sunny?]</p>

This way one-line variable declarations are succinct. Also solves the problem that "text blah blah [editorial comment]" doesn't attempt to be rendered as if [editorial comment] was a reference to variables.

{monster,frame
`[_]=@https://api.open5e.com/monsters/adult-black-dragon/`

## [_.name]
*[_.type], [_alignment]*
___
**Armor Class** :: [_.AC]
**Hit Points**  :: [_.HP]
**Speed**       :: [_.speed]

etc
}}

[ericscheid]

Then again, wrapping the variable definitions in anything could simply be unnecessary. [linkref] declarations are distinct by starting on a new line with a first part wrapped in square brackets. Variables could do the same.

[somelink]: https://example.com/
[somevar]= <some expression producing a value>

[ericscheid]

There might be a role for variables/calculations to play in determining the text of page-footers (which would normally be copying the text from the h1 from any immediate preceding {{part}}.

#2417 (comment)

[dbolack-ab]

Proposal

• Variables

  • Variables are duck-typed
  • Assign variables from a json document
  • Assign/override variables via static assignments
  • Assign/override values via math.js formulas **
  • Load variables from a remote json document
  • Variables may be local to the Expression block or persist through the brew render.
  • Stored in Marked.defaults.markedVariables if persistent.

• Assignments

  • Format:

    • =[@]+name: value|(value)|{value}|URL
    • Example:
      • =MyName: Bob

  • Variations:

    • If name is _ the assignments are local to the expression
    • If name begins with @ the value is expected to be a URL that provides a json document or a json document.
      • =@jim: https://myserver/endpoint
      • =@bob: { name: bob }
    • If value is wrapped in parenthesis it is expected to be math parsable via math.js

• Expressions

  • Inline wrapped in %

  • Inline Expressions may only be a variable, math expressions, or macros

  • Inline expressions MAY be used in mustaches.

  • Multi-line Expressions wrapped in %%

  • Multi-line Expressions may be any number of variable assignments and Inline expressions

  • Macros

    • :macro variable
    • Starting macros.
      • :inc value
        • Increment value and output
      • :dec value
        • Decrement value and output
      • :inca value
        • Output value and increment
      • :deca value
        • Output Value and decrement
      • :pageNumber offset numbering_style
        • Output the current effective page number, offset by offset and formatted via one of a stock set of formatting styles (Arabic, Roman, Appendix hyphenated, etc.)
      • :attr attribute name
        • Output the attribute from the block being modified ( only in mustache )
          • Example [foo](HTTP://myimage) {shape-outside:URL(%:attr src%)}

** Or other suitable library. It was the first I spotted that would let us feed variables into it and parse a string formula.

[dbolack-ab]

On the Macros: These are the most obvious cases I could think of for specific manipulation patterns we might do with variables that are not just math and probably can't be done or shouldn't be done by a snippet.

The :pageNumber is my best example there. I'm sure you could do some CSS tricks to get this result, but I think this works best.

Imagine we have a layout, we want all the pages up to the first page of text ( cover page, colophon, ToC, etc. ) numbered with lower case roman numbers. We want the first page of "text" to be page 1, and then the three appendices to be numbered Appendix Letter - page number of index ( IE, the first page of Appendix A is A-1. The third of Appendix C is C-3 )

I feel like there's also a place for string formatting, if nothing else, for math precision rendering.

[ericscheid]

The :pagenumber macro is indeed a desirable one, true; but it is also a wicked problem in that the actual page number as presented by the browser is also dependent on css-rules and css-counters. The latter is not something that is generally accessible to javascript, sadly.

BTW, expressing css-counter as roman numerals is already part of the css spec:
.item:before {content: counter(item-counter, lower-roman) ". ";}

[dbolack-ab]

Right, you can do some counters, but I am unaware of a way to concatenate.

[ericscheid]

I'm guessing this would work:
.item:before {content: counter(appendix-counter, upper-roman) "-" counter(appendix-page-counter) ;}

[dbolack-ab]

This seems largely incompatible with existing plans.

[G-Ambatte]

I had a thought the other day that we could use a system not unlike Ye Olde BASIC variable types:

String Definition:

[$VAR][**String Variable Here**]
[$VAR]

returns: String Variable Here

I imagine that when the Markdown Tokenizer runs, it creates a stringVariables object and pushes the variable definition's tokenized string value to a key with the variable definition name. When the Tokenizer strikes a variable call, it outputs the tokens from the stringVariables object with the same key name.
By performing this replacement during the Tokenization, this should retain the ability to overwrite the variable later in the document - doing so during Rendering would instead result in the last defined value being inserted throughout the entire document.

---

Mathematical Definition:

[# ][ x = 3 ]
[#VAR][ 2x ]
[#VAR]

returns: 6

This would operate in a similar fashion, but rather than tokenizing the value, it would instead be parsed by something like expr-eval.
This evaluation should be performed during Tokenization for the same reasons as above.

---

Neither of these can happen until #3089 is merged, because until that time, each page is it's own Markdown space, so variable information doesn't carry across a page break.
The alternative would be to move variables into the meta space, but from previous discussions this is undesirable.

---

I feel like the scope has increased immensely from the original "simple text substitution" - we don't need a sledgehammer to crack this walnut; we can always start with a simple system that we can actually implement, and extend it later to be API-querying, JSON-fetching, all-singing all-dancing Grand Poobah of variable implementations later, as the need for those features arises.

[dbolack-ab]

Neither of these can happen until #3089 is merged, because until that time, each page is it's own Markdown space, so variable information doesn't carry across a page break.

I thought that the global markedjs inclusion persisted and therefor a .use({variable:value}) persisted?

[ericscheid]

Noted for later:

expr-eval

Parses and evaluates mathematical expressions. It's a safer and more math-oriented alternative to using JavaScript’s eval function for mathematical expressions.

It has built-in support for common math operators and functions. Additionally, you can add your own JavaScript functions. Expressions can be evaluated directly, or compiled into native JavaScript functions.

I especially like that we can add our own functions into the eval sandbox. We could do things like add a custom function for lastHeading2() to get the last level 2 heading prior to the call (good for footers!). Not saying it will be easy though, I'm sure there will be tricky bits.

[dbolack-ab]

Much of what follows makes assumptions based on #3159 and #3144 being merged or very similar work taking place.

I think it would be helpful to make some small language breaks. I propose for purpose of this discussion we break "variables" up into three categories:

• User Brew Variables - This is the bulk of what we've been talking about here. These are user variables instantiated "inside" the brew document markdown.
• Document Brew Variables - These are the metadata values for the brew and the specific page currently being parsed.
• links/reflinks - Traditional link/reflink structures as marked does by default.

I think now is the time to firm up the syntax choices.

The two most discussed models I see are:

• Overloading the Link/Image/Reflink model to include declaration and expression of User Brew Variables and expression of Document view variables.
  The Upside of this is potential lower learning curve for the end user by following and extending established Markdown. The downsides are that it may require reimplementing the tokenizer for link related token capture and it may paint us into a corner for more complicated assignment constructs down the road.
• Mimicing the above with a new token delimiter, such as `[[ ]]`` as I have done in my PR. The upsides of this is it minimally impacts existing link parsing and expression and allows a bit more freedom in expanding the token pattern(s). The down side is it is yet another token.

The biggest problem ( besides a refactor :)) I see in the first option is that the only way I have been able to assign a string using the flink model is using [variable](uri title) where title can be a multi-word string. If there is another pattern without the extraneous uri, I'm all ears. Otherwise, we need to add a pattern and hopefully implement it without having to redo the full tokenizer.

The first option also makes it harder to implement inline multi-expression token and/or multi-line token. An acceptable workaround would be to drop inline multi-expression tokens and create a block level token that permits one expression to a line. e.g.

[
$[a]here
$[b]there
]

Another suggestion I've seen was to prefix the [] with $ similar to how images use !. This would help mitigate the rework potential though it is still technically "new delimiter" just in a better disguise.

---

Working through writing this I'm leaning towards a the new token idea, but using $[] instead of [[]]. Wrap assignments in parens $[variable](assignment) and ditch multi-line blocks until we decide to do math operations. For Macros, we can use a different prefix (such as >) and pass parameters in the parens.

[ericscheid]

What macros do you have in mind?

[dbolack-ab]

Honestly, I think mostly formatting things at this point. Maybe a ++ and --.

[dbolack-ab]

For example. I might track the current chapter and update it only at the top of the next for various insertion points in the footer.

[5e-Cleric]

Well this took a nice turn, variables are here, the discussion will keep open for future issues, see #3199 for the merged version.