Semantic code highlighting via Language Server Protocol

[mosra]

Pygments is great and all, but it can only do so much to highlight the code -- and especially in typeless languages (Python) or insanely hard to parse languages (C++) it can't provide type links, "go to definition" or anything like that. For C++ there's Clang-based and Pygments-compatible https://github.com/Oberon00/synth, but it seems abandoned. For reference, this is how a code file highlighted using it looks like, with references to documentation (although using outdated links in certain cases) and variable and function declaration cross-linking:

• https://tmp.magnum.graphics/ViewerExample.cpp.html

I think Language Server Protocol could help in this area -- it could be enough for m.css to pretend it's an editor and ask the server what each and every token is.

Existing implementations:

• for C++: https://clang.llvm.org/extra/clangd/
• for Python: https://github.com/palantir/python-language-server
• for Latex: https://texlab.netlify.com/ 😱
• many more: https://langserver.org/

There's also Stack Graphs by GitHub, especially of note is the "Why aren’t we using the Language Server Protocol (LSP) or Language Server Index Format (LSIF)?" question at the end, which isn't really answered in the slides. I suppose the answer is because they need it to be incremental. LSP is anything but incremental.

Referenced from there is LSIF but as far as I can see, it's currently implemented as a Node.js tool. Thus unwanted, I'd rather call into a random language-specific server directly than to have to create a server and then in addition call this Node.js tool to give me a JSON dump to further process. Or it's possibly even worse according to microsoft/lsif-node#134, with LSIF being tied to particular server implementations!? As far as I can tell, there's currently no way to make an arbitrary server create some kind of a dump on its own, so querying directly it is.

Related project (although in TS): https://shikijs.github.io/twoslash/

[marzer]

CC @marzer (just tagging myself here for future reference)