future_promise that dies

[r2evans]

This is related to

• Catch terminated/killed workers #583
• "Error in unserialize(node$con) : MultisessionFuture () failed to receive results from cluster RichSOCKnode" error - please help #584
• setTimeLimit(): varying results in multicore with failing job #665
• slow queueing of tasks when n > workers #293

It is complicated slightly by the use of the promises package, so I cannot just call value to regain the lost forked process.

I am using promises to orchestrate a multicore scenario: when I set (say) workers=20, I need to be able to queue up more than 20 without freezing the main process. I might try to keep track of this manually, or I can use promises::future_promise to do the orchestration for me.

Reprex:

library(future)
library(promises)
plan(multicore, workers=2)

pidf <- "/tmp/quux.pid"

nbrOfFreeWorkers()
job <- future_promise({
  writeLines(as.character(Sys.getpid()), pidf)
  Sys.sleep(3)
  sample(100, 1)
}, seed=TRUE) %>%
  then(
    onFulfilled = function(val) message("fulfulled! ", val),
    onRejected = function(err) message("rejected! ", conditionMessage(err))
  )
Sys.sleep(0.5)
ps::ps_kill(ps::ps_handle(as.integer(readLines(pidf))))

This is emulating a real-world use-case where occasionally the child process dies unexpectedly.

Unsurprisingly, when I ps_kill the process, I am passed the error:

rejected! Failed to retrieve the result of MulticoreFuture (<none>) from the forked worker (on localhost; PID 3572664). Post-mortem diagnostic: No process exists with this PID, i.e. the forked localhost worker is no longer alive
Unhandled promise error: Failed to retrieve the result of MulticoreFuture (<none>) from the forked worker (on localhost; PID 3572664). Post-mortem diagnostic: No process exists with this PID, i.e. the forked localhost worker is no longer alive

but then checking the number of free processes, it is still decremented:

nbrOfFreeWorkers()
# [1] 1
# Warning message:
# In mccollect(jobs = jobs, wait = TRUE) :
#   1 parallel job did not deliver a result

I have found that generally (I cannot yet say "always") I can run resetWorkers(plan()) to regain this:

resetWorkers(plan())
# NULL
nbrOfFreeWorkers()
# [1] 2

but in unsupervised use, when I may not know exactly when a child process dies, I may need to try to harvest failed workers preemptively. For that, resetWorkers errors.

resetWorkers(plan())
# Error: ‘usedCores() == 0L’ is not TRUE

I can easily catch this (try, tryCatch).

Questions:

1. Is there risk in preemptively/periodically calling try(resetWorkers(plan()), silent=TRUE)?
2. Is there a better way to know (programmatically) when a child dies and I should reset workers?
3. Is there a better way to orchestrate such that more than workers tasks does not freeze the main process, and I have better/easier awareness and control over the situation of failed processes?

[HenrikBengtsson]

Hello and thanks for the report. Background: Robustness against terminated R parallel process is still on the roadmap for Futureverse. As you've noticed, there are many feature requests around this topic. There are lots of things that need to be in place to handle this, e.g. we need to figure how it should work if a backend does not support it. Then there will be requests for automatically retrying, etc. For now, we "only" detect failed workers and throw FutureException errors. The go-to answer to what should I do when a parallel worker crashes, is unfortunately, "This means that the current R session and its workers should be considered being in a bad shape, and then the best bet is to restart with a fresh setup". If it keeps crashing on you,

That said, I think in the case of multicore futures, which are transient, forked R processes, I think it would be safe and possible to "revive" the worker slots that the crashed worker occupies. I've created Issue #677 to track this. FWIW, it might also be possible to (truly) revive a crashed multisession worker automagically. I've created HenrikBengtsson/parallelly#102 as a first step toward that.

FWIW, there's work like mirai, for which there will be a future.mirai backend, that, part of the core framework, attempts to autostart parallel tasks when they fail. future.mirai is "around the corner" - we just need mirai to settle in and find it's true identity as it's under heavy, active development right now.

[r2evans]

mirai sounds interesting, though I'd have to be careful with auto-retry: some failures self-heal, some don't, so I'd need to impose logic to prevent an unintentional DOS (or lockout or similar). I think #677 is likely the way forward for me for now, until we re-architect how I'm using promise+future in this service.

FYI, periodic resetWorkers(plan()) is not working (with try), it still hits a point where workers are not regained. For now, I'm investigating why the failures are occurring in the first case (many likely due to our previous discussion in #665, possibly most or all).

Thank you!

[HenrikBengtsson]

... I'd have to be careful with auto-retry: ...

Yes, this is really complicated. Imagine if your future runs some bad code that brings down the host (such code exists), and then you keep retrying - you'd bring down a whole compute cluster like that in a short time without even noticing.

FYI, periodic resetWorkers(plan()) is not working ...

I think this will be automatically resolved by #677.

[HenrikBengtsson]

UPDATE: The develop version of future, part of next release, now releases "crashed" multicore futures.

[r2evans]

I was just looking at that. I noticed that the future.plan=cluster test is failing, but since I'm using plan(multicore) (and that test passed), I'm likely to test the dev version and see if the problem persists. (Unless you see problems where the failed test suggests caution.)

Thank you!

(I'll keep this open while testing in case I see issues with the patch.)

[HenrikBengtsson]

FWIW, I've just updated future.callr (develop version) so that "crashed" callr futures are also fully removed.