Proposals for file system implementations

[samuelstroschein]

Problem

Inlang needs to define a filesystem API that works in server and browser contexts.

Inlang used two browser filesystem implementations already. Lightning fs and memfs. Both come with downsides that lead to this discussion.

Lightning fs limitations

• async methods don't work (bug; blocks the main thread).
• doesn't run on the server (making testing hard)

memfs limitations

• only runs in the browser with node polyfills
• hidden control flow by calling process.cwd() to resolve paths.
• type's not matching with node:fs/promises
• no caching in the browser

[samuelstroschein]

Requirements

Use this thread to discuss requirements. This post will be updated as the discussion evolves.

[samuelstroschein]

• lazy loading of files (likely git-sdk only requirement)
• cache possibilities in the browser and on the server (on the server = write to disk :D)
• runs on the server and in the browser without polyfills (no node dependencies)
• async (to not block the main thread)
• sane and spec'ed path resolution (such a big issue right now)

[samuelstroschein]

I just realized that we need to differentiate the environments. Most features are environment bound. See #485 (comment)

[samuelstroschein]

Idea: Plugin based filesystem

The requirements are so diverse that a filesystem with plugins might be the way to go. Just a sketch:

// browser fs
const fs = createFs({ storage: localStorage, plugins: [
  gitLazyLoading(),
]})

// server fs
import * as nodefs from "node:fs/promises"

const fs = createFs({ storage: localFilesystem(nodefs), plugins: [
  gitLazyLoading()
]})

// vscode fs
import * as vscodefs from "vsce"

const fs = createFs({ storage: vscodefs, plugins: [
  gitLazyLoading()
]})

Pros and Cons

✅ flexible
❌ low DX. it is fundamentally easier to just use the environment's filesystem directly
❌ breaks the "filesystem contract". files in that are read by the environment's filesystem work universally across different apps. if an app use a "special filesystem" with plugins, that contract is broken. Every app would need to use the "special filesystem".
❌ complex

[samuelstroschein]

I think the API schema and mappings idea is better #485 (comment)

[samuelstroschein]

Idea: Define a API schema and provide mappings

This is kind of what we are doing now but with wrappers instead of trying to force everything into the correct types.

Defining a minimal filesystem API. Here is the current implementation https://github.com/inlang/inlang/blob/main/source-code/core/src/config/environment-functions/%24fs.ts. Note how it uses memfs which leads to type issues.

// minimal fs that is required for inlang and the git-sdk
type FsApi = {
  readFile: (uri: string) => string
  writeFile: (uri: string, data: string) => string 
  // ...
}

Example wrapper for filesystems. Here is one in production https://github.com/inlang/inlang/blob/main/source-code/ide-extension/src/utils/createFileSystemMapper.ts.

// wrapper for node (or vscode) fs

function nodeFs(fs: fs): FsApi {
  return {
    readFile: (uri) => {
      fs.readFile(uri, { encoding: "utf-8"} )
    },
    writeFile: (uri) => {
      fs.writeFile(uri, { encoding: "utf-8"} )
    }
  }
}

We can export filesystems suited for different contexts

// example on the server
import fs from "node:fs/promises"
import { nodeFs } from "@inlang/fs"

// wrapping the API around node's fs.
clone({ fs: nodeFs(fs) })

// example for vscode
import fs from "vsce"
import { vscodeFs } from "@inlang/fs"

// wrapping the API around node's fs.
clone({ fs: vscodeFs(fs) })

// in the browser (localStorageFs is a custom fs implementation by us).
import { localStorageFs } from "@inlang/fs"

clone({ fs: localStorageFs })

// for testing purposes 
import { memoryFs } from "@inlang/fs"

clone({ fs: memoryFs })

✅ Obeys to the "filesystem contract". Other apps can use the environment's filesystem as expected.
✅ use whatever filesystem is most suited for your app as long as the filesystem obeys to the API schema, or a wrapper exists.
✅ Simple.

[samuelstroschein]

Moving a discussion from Discord (link) @araknast wrote:

Definitely agree with the api approach opposed to plugins, then fs support for vscode and node environments is trivial. That leaves the browser environment, for which I see 3 main approaches:

1. Further develop LightningFS, this could be fairly time consuming, but would most likely result in the fastest and lightest possible fs implementation.
2. Use BrowserFS, larger and slower than than lightningFS, but doesn't seem to suffer from the same issues, as well as being extremely flexible in terms of backends (XHR, LocalStorage, FileSytem, Dropbox, IndexedDB, etc.)
3. Write a basic in-memory filesystem from scratch, probably the slowest solution, but also the lightest and easier than further developing LightningFS. The largest concern besides speed is memory usage of course, but this should be alleviated by lazy loading.

The approach we take depends on the needs of the editor, curious what you think

[araknast]

Do you think the desire to iteratively develop a filesystem based on emerging requirements outweighs the cost to develop a basic in-memory filesystem at first?

I think so, I agree that it should be fairly trivial to develop a basic in-memory filesystem, so even if it doesn't work out it's not much dev time lost. I'll try and get some code up today or tomorrow and we can go from there.

[samuelstroschein]

@araknast sounds good! I propose a new module @inlang/fs that can be used by the git-sdk and inlang core (is not coupled with the git-sdk per se)

[araknast]

Here's what I was thinking of for a minimal fs implementation. Will do further testing with the editor tomorrow or later on today, in the meantime let me know what you think / what could be improved

[samuelstroschein]

Looks good! A few remarks:

• the inlang editor only needs utf-8 string file right now. exposing arraybuffers might be a thing for the future
• we need toJson and fromJson functions
• returning a map for a directory seems to go against the intuition of expecting an array with a list of paths. plus, a map can't be iterated on with for of loops etc.

I thought I help you out and adjusted the filesystem API/schema and lay out the initial file structure. You can directly commit to the https://github.com/inlang/inlang/tree/%40inlang/fs branch, specifically the inlang-source-git directory. I created a loom that explains the structure https://www.loom.com/share/65b88e3bfe6e4e1cab8f95fab2348dcd.

I propose that we switch review processes to PRs and you get your hands dirty by working directly in source-code-git :)

[james-pre]

Hi @samuelstroschein, I'm the maintainer of BrowserFS. I'd like to let you know that BrowserFS has been deprecated. You can read more about that on the BrowserFS readme: https://github.com/jvilk/BrowserFS. ZenFS is an updated and active fork that I created.

[samuelstroschein]

Should we go for OOP or a functional/object-based implementation?

The discussion in #540 (comment) revealed that an open question is whether an OOP or functional approach leads to better ergonomics.

Oh, I had the "schema" in mind for the universal API "schema" for all filesystem implementations, see #485 (comment).

If fromJson is not part of the "schema", there is no guarantee that fs implementations will provide a fromJson function. The code below is a pseudocode example where fs needs to provide fromJson and toJson methods as part of the "schema".

async function loadConfig(fs: Fs) {
  const config = await fs.readFile("./inlang.config.js")
  // immatuable/new filesystem
  const copy = fs.fromJson(fs.toJson())
  await copy.writeFile("./inlang/metadata/config.js", config)
  return copy
}

Now, I don't know yet whether the need for copying a filesystem/having the fromJson method will emerge, but, the simplicity of having a type that defines the interface without static and other OOP styles might be worth considering due to simplicity and tree-shaking capabilities. I'll open a discussion.

Benefits of a functional approach

One type defines the entire interface

Using a functional approach allows us to define a schema like:

type Filesystem = {
  readFile: (path: string) => Promise<string>
  fromJson: (json: string) => Filesystem
  // ... 
}

An OOP approach with static methods would prevent fromJson to be defined in the interface and, more importantly, inlang expects the filesystem to be provided by the host application (web editor, ide extension, or CLI). Using an OOP approach would make the following impossible:

async function loadConfig(fs: Fs) {
  const config = await fs.readFile("./inlang.config.js")
  // copying the filesystem with the fromJson method.
  const copy = fs.fromJson(fs.toJson())
  await copy.writeFile("./inlang/metadata/config.js", config)  
  return copy
}

The consuming function loadConfig can't import the right filesystem that corresponds to the toJson method e.g. MemoryFs.fromJson() or NodeFs.fromJson()?

Functional approach can be tree-shaken/use dynamic imports to reduce bundle size

Since the filesystem API will be async only, we can use dynamic imports to reduce the bundling size of the filesystem.

While this argument doesn't matter with a few functions, a complicated filesystem implementation with heavy caching capabilities (for example) and lots of functions likely benefits from a functional approach. A functional approach should ease bundle size reductions and performance optimizations. For example, bundlers can tree-shake functions. The dynamic import pattern showcased below also feels more natural in a functional approach than OOP.

function createMemoryFs(): Filesystem {
  const files = []
  // other state

  // all methods are lazy loaded
  return {
     readFile: async (...args) => (await import("./readFile.js")).default(args)
     fromJson: async (...args) => (await import("./fromJson.js")).default(args)
     // ...
  }
}

Simpler composability

Composability seems simpler if a functional approach is chosen.

For example, I imagine following code patterns to eventually merge that share code between implementations. While yes, OOP can achieve a similar effect, defining a well, and independently, tested lazyLoad function once seems easier than relying on class inheritance. Regardless if this example is the right one, composability with functions seems easier.

function createNodeFs(): Filesystem {
  return {
    readFile: lazyLoad(fs.readFile)
    // ...
  }
}

function createMemoryFs(): Filesystem {
  return {
    readFile: lazyLoad(readFile)
    // ...
  }
}

Writing reactivity wrappers likely becomes easy too:

function withSolid(fs: Filesystem): Filesystem {
  const retrievedFiles = createStore({})
  
  return {
    readFile: async (path: string) => {
       const file = await fs.readFile(path)
       setStore(path, file)
       return retrievedFiles[path]
    }
  }
}

(The rest of the code base uses the functional approach)

Minor but the rest of inlang uses the functional approach.

[samuelstroschein]

Using an instance method prevents the following line of code to work:

async function loadConfig(fs: Fs) {
  const config = await fs.readFile("./inlang.config.js")
  // immatuable/new filesystem
+ const copy = fs.fromJson(fs.toJson())
  await copy.writeFile("./inlang/metadata/config.js", config)
  return copy
}

I don't know if the use case emerges but returning a new file system / treating fromJson as static factory method enables a new use case. But, this use case might be heavily targeted towards a memory filesystem. Returning a new instance of nodeFs, for example, seems to make no sense. fromJson should update the "node fs" in place.

TBH we can go either way. Let's go with fromJson which updates the fs object in place. If the use case emerges for duplicate, we can add duplicate at a later point in time.

@araknast agree/disagree?

[atk]

Having to represent the whole filesystem as JSON in memory to do a copy seems pretty wasteful. I'm currently preparing an rsync to allow copying recursively between filesystems for my own primitive, which uses recursion to walk the directories to be copied. I hope to have it finished next week. JSON is fine if you need to persist a filesystem as string or send all of it somewhere, but for local copying, it's a worst-case scenario.

[samuelstroschein]

JSON is fine if you need to persist a filesystem as string or send all of it somewhere, but for local copying, it's a worst-case scenario.

Sending the filesystem (toJson) over the network (and rebuilding the filesystem with fromJson) is what we need. An example of real-world usage can be found here https://github.com/inlang/inlang/blob/main/source-code/rpc/src/functions/generateConfigFile.ts#L60.

I take that you advocate for an in-place mutation with fromJson (correct me if my impression is misguided). I will start to rip out memfs with the implementation we currently have to see what blockers emerge and to unblock one of my tasks.

[araknast]

TBH we can go either way. Let's go with fromJson which updates the fs object in place. If the use case emerges for duplicate, we can add duplicate at a later point in time.

@araknast agree/disagree?

Agreed, this approach makes the most sense to me. Also agree that serializing/de-serializing the filesystem to create a copy is inefficient, which I think is a good argument for having duplicate as a separate method that copies the objects directly if necessary.

[atk]

I had more time on my hands than I had anticipated and thus already finished the rsync utility: solidjs-community/solid-primitives#407 - if you restrict it to non-reactive file systems, you can lose some of the code, too. Maybe this helps you.

[atk]

I had tested an OOP approach with an ORM-like implementation for @solid-primitives/filesystem, but that really became unintuitive and unwieldy, especially in combination with asynchronous handling. Also, filesystem names rarely make good property names, so you will either need filtering and introduce ambiguity or use brackets everywhere.

My advice would be to go with a functional approach.

[samuelstroschein]

Thanks for the input!

[samuelstroschein]

We decided on a function approach that is a subset of node:fs/promises.