Snippet representation

[pokey]

We should come up with a good representation for our snippets. Currently they look like the following:

{
    "classDeclaration": {
        "definitions": [
            {
                "scope": {
                    "langIds": [
                        "python"
                    ]
                },
                "body": [
                    "class $name:",
                    "\t$body"
                ],
                "variables": {
                    "name": {
                        "formatter": "pascalCase"
                    }
                }
            },
            {
                "scope": {
                    "langIds": [
                        "typescript",
                        "typescriptreact",
                        "javascript",
                        "javascriptreact"
                    ]
                },
                "body": [
                    "class $name {",
                    "\t$body",
                    "}"
                ],
                "variables": {
                    "name": {
                        "formatter": "pascalCase"
                    }
                }
            }
        ],
        "description": "Class declaration",
        "variables": {
            "body": {
                "wrapperScopeType": "namedFunction"
            }
        },
        "insertionScopeType": "statement"
    }
}

This representation is hard to read

[pokey]

I was thinking something like the following, using toml preambles:

---
id = 'classDeclaration'
description = 'Class declaration'
insertionScopeType = 'statement'

[variables.name]

[variables.body]
wrapperScopeType = 'namedFunction'
---

---
id = 'classDeclaration'
langIds = ['python']

[variables.name]
formatter = 'pascalCase'
---
class {{ name }}:
    {{ body }}

---
id = 'classDeclaration'
langIds = ['typescript', 'typescriptreact', 'javascript', 'javascriptreact']

[variables.name]
formatter = 'pascalCase'
---
class {{ name }} {
    {{ body }}
}

Note how in the first "snippet", the body is empty, so we're just defining attributes without defining a snippet body, and we don't use langIds, which makes it global

Note also that we strip trailing newlines in the snippet body so that the user can add extra spacing for readability

Looks like there is a tree-sitter-toml, as well as a cool pre-release toml extension that supports schemas, and the engine it uses is also available standalone as a JS package using wasm yarn add @taplo/lib

Looks like with that toml extension we can signal important keys in the schema via initKeys so that it will automatically add them in autocomplete

At its simplest, a snippet could just look like:

---
id = 'helloWorld'
---
Hello, {{ name }}!

[wenkokke]

What would the main advantage of using TOML will be?

[pokey]

nunjucks looks kinda nice as a templating language for the body of the snippet.

I like the way the filters look; a lot like jq

Has a few VSCode extensions but this one gets the best reviews it appears

[pokey]

We'd need to figure out how to handle the VSCode special variables and variable transforms

[pokey]

We also need to think about whether the looping syntax for iterating vector snippet holes (#445) is semantically rich enough for us to be able to keep track of them as they grow when used in #443

[pokey]

Fwiw here is a list of template engines

[pokey]

Also fwiw here is how it would look if we used yaml for the metadata header:

---
id: classDeclaration
description: Class declaration
insertionScopeType: statement
langIds:
- python
variables:
  body:
    wrapperScopeType: namedFunction
---

---
id: classDeclaration
langIds:
- python
variables:
  name:
    formatter: pascalCase
---
class {{ name }}:
    {{ body }}

---
id: classDeclaration
langIds:
- typescript
- typescriptreact
- javascript
- javascriptreact
variables:
  name:
    formatter: pascalCase
---
class {{ name }} {
    {{ body }}
}

if we go with yaml we'd want to register schema like this

[wenkokke]

Wouldn’t it be easier to use multiline fields for the actual body, rather than separating YAML headers from non-YAML bodies? The only price to pay is that the body must be indented, but for that we gain that we do not have to write any parsers. Furthermore, it comes with a clear and unambiguous way to move onto the next snippet: dedenting.

[wenkokke]

Like so:

---
id: classDeclaration
description: Class declaration
insertionScopeType: statement
langIds:
- python
variables:
  body:
    wrapperScopeType: namedFunction
...
---
id: classDeclaration
langIds:
- python
variables:
  name:
    formatter: pascalCase
body: |
  class {{ name }}:
      {{ body }}
...
---
id: classDeclaration
langIds:
- typescript
- typescriptreact
- javascript
- javascriptreact
variables:
  name:
    formatter: pascalCase
body: |
  class {{ name }} {
      {{ body }}
  }
...

[pokey]

yeah it would be nice not to need to write a parser 🤔. I am of two minds here because yaml can be a bit annoying to write, given how ambiguous it is. But I agree that if we go the yaml route then maybe it's best to leverage its multiline text niceties

[wenkokke]

Personally, I think that the YAML used in our snippets probably won’t (or shouldn’t) escalate to any size or complexity where YAML becomes maddening. Plus, it has as an advantage that our existing snippets are already valid!

[wenkokke]

Also, I guess you personally will be creating snippets automatically anyway, so?

[pokey]

Do we want to use --- both for separating snippet metadata from body as well as for separating the snippets, or use a different separator?

[pokey]

Could maybe use === for separating snippets and --- for separating header from body. Could also drop the initial ===?

[AndreasArvidsson]

I definitely prefer different dividers for this

[wenkokke]

If we’d use YAML, it defaults to --- and ..., so maybe we could adapt something similar with TOML headers?

[pokey]

with yaml, do you even need the ...? I thought you just need --- to separate documents

[pokey]

could be a bit confusing the fact that the different bodies are associated just by the fact that they share an identifier. Could be a bit confusing / error prone. Possibly just have it so that if you only specify identifier in first snippet in body, the rest of the snippets in the language don't need to specify the identifier and are assumed to have the same identifier

[AndreasArvidsson]

Do have any thoughts on how to handle tabs/indentation now when we represent the body as plaintext?

class {{ name }}:
    {{ body }}

Is that a tab symbol? four spaces?

The reason why this matters is that vscode handles the insertion of a tab symbol according to the users tab settings, but if the body actually contains just white spaces as indentation I don't believe that actually happens.

[wenkokke]

I guess we don’t actually have access to the users tab settings? We could allow the user to configure the number of spaces that is meant to be interpreted as an indent as part of the metadata, and try to have sensible defaults? The exact number of spaces that you’re indenting might be relevant in a whitespace sensitive language such as Markdown or Haskell.

[wenkokke]

A decent auto mode could be: if there’s any tabs, tabs are indents; if there’s no tabs, check the size of the first indent and use that to determine the number of spaces per indent (either 2 or 4). Any trailing spaces (either in tab mode or those not part of a whole indent) are inserted verbatim.

[pokey]

I think my algo linked above should work for most cases, though I guess markdown is an interesting one, because if you are in a list, then you need 8 spaces to create a code snippet, tho I don't think your solution would work in that case either.

I'd be inclined to use my linked solution, but then have a flag in the metadata that indicates to treat indentation literally

[wenkokke]

List items in Markdown in general actually may depend on the exact number of spaces as well:

Similarly, some constructs in Haskell, when split across lines, may depend on an exact indentation as well, e.g., using do-notation as follows, all actions must be aligned:

do action1
   action2
   action3

[pokey]

Understood. Does my proposed solution sound good?

[ParetoOptimalDev]

You might want to consider copying yasnippet: https://github.com/joaotavora/yasnippet

You write almost no more than the thing you are templating!

It also looks like someone already implemented it in vs code:

https://marketplace.visualstudio.com/items?itemName=mads-hartmann.yassnippet

If you need more information than this, I'd encorage parsing that format which is very convenient for end users into a richer one.

[pokey]

Yeah there is something to be said for sticking with that flavour of syntax, as it is quite widely used. I believe @wenkokke suggested yasnippet as well. Looks like yasnippet has support for transformations, which is one of the things we were after

I'd be ok with using yasnippet, but adding some additional supported fields in the frontmatter as required for cursorless's structurally aware snippet gymnastics, such as auto-expanding to scope type during wrapping, etc

[ParetoOptimalDev]

I'd be ok with using yasnippet, but adding some additional supported fields in the frontmatter as required for cursorless's structurally aware snippet gymnastics, such as auto-expanding to scope type during wrapping, etc

This is what I had in mind, but you might be trading user convenience for a more difficult implementation.

There is an argument that concise templating syntax doesn't matter as much if most snippets are created interactively, but somehow i always find myself having to manually do something .

[wenkokke]

You might want to consider copying yasnippet: https://github.com/joaotavora/yasnippet

You write almost no more than the thing you are templating!

It also looks like someone already implemented it in vs code:

https://marketplace.visualstudio.com/items?itemName=mads-hartmann.yassnippet

If you need more information than this, I'd encorage parsing that format which is very convenient for end users into a richer one.

Yes, I was also advocating for using the yasnippet format! It is pretty much my favorite format, but I think that the main downside for @pokey is the lack of support for multiple snippets in a single file. Personally, I also almost always end up writing this snippets myself, rather than creating them interactively. Obviously, that might change if I really learn how to use the interactive snippet creation commands, but it might still be easier for newcomers.

[wenkokke]

I should definitely look into mads-hartmann.yassnippet. My main confusion would be that yasnippet uses the globally available elisp functions, so I don't really know how you could do this in an editor agnostic way?

[wenkokke]

Everything after the # -- marker is the body of your snippet. See vscode documentation for the syntax.

The following comment leads me to believe that what this extension actually implements is a little bit of ya snipped sauce on top of the vscode snippet engine, so only very basic snippets would be portable between editors, whereas the main features of ya snippet that we are after are the transformations.

[wenkokke]

One consideration is whether or not template engine constructs such as if-then-else and for loops can reference user input, and whether their bodies can hide/expose further holes. I think we should be able to handle such cases, but doing so would expose several UX and programming issues:

Holes may appear, disappear, and reappear at any point during snippet resolution. Do we remember their previous values? What about holes which have no absolute name, e.g., holes which are generated by a for loop? Do we just remember all values so far in a list?

From an implementation perspective, we’d have to rerun the template engine at every user edit in a hole, since the template may have resolved differently. However, the user may switch between editing the contents of holes and the text generated by the template. How do we remember these updates to the generated text. Barring for loops, we could probably use the existing cursorless functionality to dynamically create a “new” temporary snippet if we remember where the snippet started and where it ended, and we know where the holes are. However, this becomes a lot more tricky if we, say, edit the standard text generated by a for loop in only the third instance. So maybe just remembering edits to the generated text and rebasing them, as it were, would be better?

[wenkokke]

All this to say, I think we may want to implement our own tiny template engine, with a template language which doesn’t support control-flow constructs, but only tab stops with transformations.

(Also saves us that one bug report a week after release where someone points out that the template engine we settled on refuses to stop HTML escaping the snippets.)

[pokey]

Whoah ok I see you're starting to think about this stuff deeply 😅. Shall we find a time for screenshare?

[AndreasArvidsson]

Yesterday we had another discussion about snippets and we decided that Cursorless snippets actions should be able to be utilized by custom/external snippets sent over the command server.
#1325

My goal would be to have a snippet representation that we could upstream to the community and be used by different ide and projects.

I've been thinking on a much simpler snippet format that reminds a lot about Talon files.

scopeType: statement

# Single key value pair, but we split the value later
languages: javascript, typescript, javascriptreact, typescriptreact

# Actual support for lists
languages: 
    javascript
    typescript
    javascriptreact
    typescriptreact

# Divider between header and body
-
try {
    $try
}
catch(error) {
    $catch
}
---

scopeType: statement
languages: python
-
try:
    $try
except Exception as ex:
    $catch

[wenkokke]

If this is what you want to write, why not use YAML, which could write almost exactly that, but wouldn’t need a custom language spec or parser?

[wenkokke]

For compatibility with the Talon format, you could just repeat the language keywords once per language:

scopeType: statement
language: javascript
language: typescript
language: javascriptreact
language: typescriptreact
-
try {
    $try
}
catch(error) {
    $catch
}
---
scopeType: statement
language: python
-
try:
    $try
except Exception as ex:
    $catch

[wenkokke]

How does this denote indentations?

[AndreasArvidsson]

Because yaml is a complicated format and I don't want the users to have to learn yaml.

I thought about repeating keys. It is an easy format to parse, but quite ugly to my eyes.

I was probably going to just parse the body and calculate indentations.

[wenkokke]

Anything other than repeating keys would deviate from the Talon format, though.

[AndreasArvidsson]

After some thorough thought I have now gone ahead and implemented a format that I'm very happy with. @pokey @josharian have already given input and I have made some tweaks based on that.

https://github.com/AndreasArvidsson/andreas-talon/tree/master/core/snippets

[AndreasArvidsson]

I'm not a big fan of relying on file names to be honest.

[AndreasArvidsson]

Format updated with wrapperPhrase

[pokey]

I guess you could support that same | for insertion scope, to support fallback insertion scopes like we use with functionDeclaration

[AndreasArvidsson]

Today the format doesn't even support insertion scope because the Cursorless api doesn't. Once that is fixed we can definitely support multiple scopes in the snippet.

[pokey]

Oh really? Was sure I saw a scope at the top but must have been confused. Yeah makes sense