Directed Graph Visualization of Zettelkasten

[DevonMorris]

It would be useful to have an overview of the zettelkasten viewed as a directed graph. I was thinking d3 could be used to make this visualization.

Something like https://observablehq.com/@d3/force-directed-graph. Coloring could be performed based on tags.

This would give an overview of the topology of the zettels and could be used to find useful or missing connections.

[srid]

@aca had a proof of concept version of something like this here: https://aca.github.io/neuron_zettels_in_3d.html

In any case, neuron exposes the graph as JSON - so you can do a little scripting to visualize it however you want. See https://neuron.zettel.page/cache.json

That said, yes - it would be great if neuron included a graph view that is both direction-aware and folgezettel-aware.

[DevonMorris]

Oh I wasn't aware of that cache.json! Thanks for pointing it out!

[DevonMorris]

https://devonmorris.dev/graph.html It's far from perfect, but I was able to whip something up pretty quick

[srid]

Looks pretty cool! How far you can customize it? For eg: a) add links to html pages, b) create a more legitimate layout rearranging nodes automatically?

I'll have to checkout d3 sometime.

[DevonMorris]

a) If you double click any node, it should already take to to the corresponding zettel. I need to add some text to the page for the events/bindings that I have D3 listening to. For non-zettel/external html, this will be much harder since I'm pulling from that cache.json.

b) https://github.com/d3/d3-force there are lots of parameters here to mess around with. I'm sure tweaking them in some fashion will yield more pleasing results.

I'm hoping to have this a bit more cleaned up in the next few days, so I'll let you know of any progress I make. I didn't do anything to distinguish folgezettel links, so that's an obvious improvement.

[DevonMorris]

Should also mention that size and color are determined by in degree. And you can manually rearrange the graph by dragging the nodes to pin them. ctrl+click will unpin them.

[DevonMorris]

Also I moved the link around this morning, so it'll be at https://devonmorris.dev/graph.html now

[jaygray0919]

@DevonMorris we were working on exactly the same idea but trailed your implementation. Can't thank you enough for this example. We'll stand on your shoulders.

Aside for the benefit of @srid and others.

The DevonMorris example implements the miserables model. There are other d3.js data models such as the flare model - the original hierarchical model used for dendrograms and sunburst diagrams.

Here is a knowledge graph that implements the miserables model using three.js https://afdsi.com/data/onto-id-v3/
It's big so tap F5 if it fails to load.

We could easily take the DevonMorris data and implement the three.js view.

@srid the issue is:

how to export neuron zettel-relationships
(in two parts: initialize the identifiers; then generate the node pairs to build the graph).

Why do this? There are two ways to express the reason:

1 Because it's not easy to do with any other comparable tools (e.g. Roam, Org Mode) "take that you swine ..." (Monty Python)

2 Because it illustrates the structured properties of neuron as a knowledge documentation engine.

@srid likes to emphasize neuron future-proof features based on MD as portable documents (application-independent).

The DevonMorris work demonstrates the interchange of a network model (based on miserables) that enables a miserables processor (d3.js) to visualize the specific data model used by a specific neuron-based blog.

Exciting stuff @DevonMorris

[jaygray0919]

@srid aside: the above knowledge graph is the data model we discussed on the teleconference several months back when asking the question:

how to import defined types and their properties into a Haskell program, and then add type-based routines (e.g. the combination [e.g. math] of values for 2+ identical type) to the program.

Another example: summation of values for an array of identical types. Practical example: summation of the value of all amino acids found in the edible portion of a food). [nearly impossible to do in any non-typed programming language]

Topic for another thread on another repo ...

[DevonMorris]

Since this has become a broader discussion, I eventually want to run some graph-based analytics on this graph information. Still deciding what the best tool to do this would be.

I'm thinking stuff like finding the diameter of the network, distance between nodes of interest, most central nodes in the network and running page rank on the network.

I'm sure there's a lot that could be done here.

[jaygray0919]

Another key marketing point here. Most of the Knowlege Graph stuff (that's the parent term for the more specific term technology) is theoretical. The DevonMorris blog is a real-world, practical example of a Knowledge Graph (implemented in one technology and visualized in a second technology). Examples like that, and our Knowlege Graph, transform abstract concepts expressed in pubs like Medium and make them tangible (and reproducible).

[srid]

If we can get this in good shape, I'd be happy to include the graph .js/.css¹ in neuron itself (along with generating graph.html), so that all other users can automatically benefit from it.

Long-term we might be able to unify both /impulse and /graph, as long as the graph view evolves to the point of representing heterarchy tree in an improved fashion (eg: resolving #8) and the JS is as performant as statically generated HTML for large zettelkastens.

@jaygray0919 Thanks for the info. If/when I get to doing some explorations on d3.js I'll comment here. I am interested more generally in visualization of (semi-)structured data for another project as well. Data is often presented linearly, which is pretty limited - compared to how the human mind visualizes objects in the real world (say, on a desk or room).

^{[1] I'm not anymore averse to using JS for isolated features like this. I think we should do it, and latter we may switch it to using more type-safe languages like PureScript.}

[DevonMorris]

@srid, I'd be happy to help with this, right now it's kinda hacked together for my purposes, so it probably needs a bit of work to become generalized in terms of theme etc. Besides the performance implications, what requirements do we have to include this the upstream?

[srid]

For performance, talk to ybaumy on Element.io group - for a 6000-note site, it does freeze the browser: https://lost-frequencies.eu/graph.html

Otherwise, I'd say for beta-level release let's put it in /graph.html and link to it from the /impulse.html page. It can be refined later; and integrated better once performance is handled in someway (for eg., larger notebooks could display only a certain level top-level folgezettels).

[samwalls]

@DevonMorris I use your graph on my notes as well. It's very handy!

I made some simple changes to the link-generation step which let you filter out certain things. It helps reduce the amount of visual noise, depending on how you personally link-up your notes. You should be able to paste this over the existing code and try it out.

    // Compute number of incoming links
    var links = [];
    Object.entries(graph.adjacencyMap).forEach(([node,others]) => {
      console.log(JSON.stringify(graph.vertices[node].Meta));
      var ignored = ['index', 'TODOs'];
      var ignoreFromTag = ['personal/log', 'meeting'];
      var ignoreToTag = [];
      var ignoredLinkType = ['cf']; // may also include 'folge' and 'folgeinv'
      if (ignored.includes(node)) {
        // explicitly ignored nodes
        return;
      }
      if (ignoreFromTag.some(tag => graph.vertices[node].Meta.tags.includes(tag))) {
        // ignore links *from* certain tags
        return;
      }
      Object.keys(others).forEach((otherNode) => {
        if (ignored.includes(otherNode)) {
          // explicitly ignored nodes
          return;
        }
        if (ignoreToTag.some(tag => graph.vertices[otherNode].Meta.tags.includes(tag))) {
          // ignore links *to* certain tags
          return;
        }
        if (ignoredLinkType.some(linkType => others[otherNode].includes(linkType))) {
          // ignore specified link types
          return;
        }
        graph.vertices[otherNode].inDegree += 1;
        var link = {};
        link.source = node;
        link.target = otherNode;
        links.push(link);
      })
    });

Might be useful to have some sort of basic filtering tools, and modes of display with different emphases, on the graph page?

[DevonMorris]

Thanks @samwalls, I got caught up in school, work and life stuff and haven't been able to look at this again. Maybe this weekend? 🤷🏻