nanonext, mirai, and clustermq

[nviets]

Clustermq is my go-to library for HPC computing in R, given its use of zeromq and schedulers like slurm. Where are nanonext and mirai headed next? Is there a natural way to extend mirai with schedulers or plug nanonext into clustermq as another method? I understand nng has some advantages over zeromq.

[shikokuchuo]

You may find interesting the existing discussion on potentially implementing a task queue/scheduler that is kept open in issue #3. The short answer is that I have not got round to it yet. Recent development has gone into areas higher on my own priority list which has been the performance/stability of nanonext 0.6.0 (to be imminently released).

In principle I am open to integrations of nanonext with other frameworks, especially if other maintainers see a need and reach out.

[shikokuchuo]

Just implemented is the ability for mirai to distribute tasks across the network. I have not implemented a convenience function for e.g. slurm, but you should be able to set a server process (or many of them) e.g. by:

Rscript -e 'mirai::server("tcp://192.168.0.2:5555")'

where 192.168.0.2:5555 is the IP and an open port on the client (requestor).
Then in your client process you just set

daemons(.url = "tcp://192.168.0.2:5555")

I have followed your discussion with @wlandau as you have copied me in. I think mirai is taking a very different approach with minimal dependencies and a lean stack. I have even resisted implementing a queue thus far (although this may come in at some point).

[shikokuchuo]

Forgot to say you want the latest dev versions of nanonext and mirai:

install.packages(c("nanonext", "mirai"), repos = "shikokuchuo.r-universe.dev")

[nviets]

Thanks for the update. I'll check it out soon.

[shikokuchuo]

The latest dev version now incorporates an active queue / scheduler. Use for example:

daemons(4, q = TRUE)

Your tasks will be queued and allocated to the next available daemon.

[shikokuchuo]

As I am sure you have been following, I have been collaborating with William Landau to make {crew} https://cran.r-project.org/package=crew possible. We are still working on optimising some of the internal workings, but ultimately that should provide you the platform to extend mirai to slurm etc.