How do I create an asynchronous source where the output is filtered by elisp code?

[matthuszagh]

I wrote a program that parses a JSON file and retrieves relevant entries (also json) based on a provided search string. I'd previously been using helm-build-sync-source to parse this and display a formatted candidate for each entry in the JSON output, which worked great up until recently. Unfortunately, the JSON file being parsed has grown quite large such that searching and retrieving relevant JSON entries takes a non-negligible amount of time (I'm measuring about 0.3s). This is enough of a delay that when I type at normal speed, some key presses are ignored. I believe this is because helm is waiting for the process to finish. When I type slowly, all key presses are registered.

So, I started looking at asynchronous sources so that the program runtime does not interfere with typing the search terms. However, there doesn't appear to be a way to transform the asynchronous process output using elisp. A similar issues was mentioned here. Unfortunately, the solution there of using a synchronous source doesn't work here. Is there a way of doing this I haven't considered?

Please let me know if this is in any way unclear. I've so far refrained from providing code because it's a bit tough to come up with a MWE in this case. But, if you'd prefer code I can try to share what I have.

[thierryvolpiatto]

matthuszagh ***@***.***> writes:

1. ( ) text/plain (*) text/html I wrote a program that parses a JSON file and retrieves relevant entries (also json) based on a provided search string. I'd previously been using helm-build-sync-source for this, which worked great up until recently. Unfortunately, the JSON file being parsed has grown quite large such that searching and retrieving relevant JSON entries takes a non-negligible amount of time (I'm measuring about 0.3s).

If your json file is not constantly modified, what about caching your json entries? See https://github.com/emacs-helm/helm-skitour.

This is enough of a delay that when I type at normal speed, some key presses are ignored. I believe this is because helm is waiting for the process to finish. When I type slowly, all key presses are registered. So, I started looking at asynchronous sources so that the program runtime does not interfere with typing the search terms. However, there doesn't appear to be a way to process the asynchronous process output in elisp.

The problem with asynchronous process is that the output is truncated in chunks of text, and json output may have no sense when it is truncated. So I am not sure it is a good solution, otherwise the output of any async source in helm can be filtered an treated with filtered-candidate function.

…

-- Thierry

[matthuszagh]

The cache idea is interesting, thanks for the suggestion.

However, I'd still like to investigate the idea of an asynchronous source if possible. From reading make-process, it sounds like candidates can be written to a buffer and then a "sentinel" can be invoked when the process completes. The sentinel could then read out the full buffer contents to helm for further processing. Is something like this feasible?

A potentially easier way (though it relies on a 3rd party libary) would be to use something like emacs-async. This can perform a callback with the process output that then invokes a helm function, etc. I expect you wouldn't want to rely on a external library, but maybe I could provide my own defclass inheriting from helm-source that does this? Do you think something like this would be possible?

[thierryvolpiatto]

matthuszagh ***@***.***> writes:

1. ( ) text/plain (*) text/html The cache idea is interesting, thanks for the suggestion.

For this you create an init function (:init slot) that create the cache (your 0.3s will be spent there) then your :candidates function will use the cache (a variable) instead of a function, you can add to your source an :update function that reinit the cache i.e. `(setq cache nil)` that add the ability to refresh the cache with `C-c C-u` if needed. You can keep the cache for a while or rebuilt it at each restart of your helm session.

However, I'd still like to investigate the idea of an asynchronous source if possible. From reading make-process, it sounds like candidates can be written to a buffer and then a "sentinel" can be invoked when the process completes. The sentinel could then read out the full buffer contents to helm for further processing. Is something like this feasible?

Yes, look in helm source code, you will find many sentinels but I guess the json parsing will then happen there, blocking as before.

A potentially easier way (though it relies on a 3rd party libary) would be to use something like emacs-async. This can perform a callback with the process output that then invokes a helm function, etc. I expect you wouldn't want to rely on a external library,

We already use emacs-async as dependency (I am the maintainer) but I think it will be slow as well.

…

-- Thierry

[thierryvolpiatto]

Also I don't know what you are using to parse your json output, but I suggest you use `json-parse-string` if you don't already do it.

…

-- Thierry

[matthuszagh]

Most of the processing time is spent retrieving relevant JSON entries using the external tool; I'm using that external process for matching instead of helm. I then use :candidates to accept the JSON corresponding to matched results and transform it into a display and real that I want. I also use helm to perform actions on these candidates. I am currently using json-parse-string to parse the matched JSON in my candidates function. I haven't profiled that, but I think almost all of the time is spent on that external process. So, I'm optimistic that an async solution + sentinel could provide some benefit.

However, I just found the :input-idle-delay setting and this works really well, so I'm not sure I really need an async source anymore.