How to avoid "detritus in the temp directory" R CMD check on Windows

[HenrikBengtsson]

Consider the following tests/mirai3.R package test:

library(mirai)

message("Reset")
mirai::daemons(0)
print(mirai::daemons())

message("Launch two workers")
mirai::daemons(2)
print(mirai::daemons())

message("Reset")
mirai::daemons(0)
print(mirai::daemons())

Sys.sleep(5)

When running R CMD check --as-cran on MS Windows, this results in:

* checking for detritus in the temp directory ... NOTE
Found the following files/directories:
  'Rscript177cbeb83'

I'm pretty certain those Rscript177cbeb83 files are leftovers from parallel workers launched in the background that was not properly shut down, or shut down in time. This happens for instance with parallel::makeCluster() workers where we forget to call parallel::stopCluster(). See Section 'For package developers' in https://parallelly.futureverse.org/reference/makeClusterPSOCK.html#for-package-developers-1 for details.

I've created a minimal R package that illustrates this problem via GitHub Actions available at https://github.com/HenrikBengtsson/teeny/tree/package/mirai.

How can I make sure that all background R processes that mirai launches are properly shut down before I exit my test script?

[HenrikBengtsson]

Maybe it's the dispatcher process that leaves this behind, or is not shut down. When using dispatcher = FALSE (see below), the "detritus in the temp directory" NOTE is gone.

library(mirai)

message("Reset")
mirai::daemons(0)
print(mirai::daemons())

message("Launch two workers")
mirai::daemons(2, dispatcher = FALSE)
print(mirai::daemons())

message("Reset")
mirai::daemons(0)
print(mirai::daemons())

Sys.sleep(5)

[shikokuchuo]

It'd be a bit odd if it were dispatcher itself, as exits are a cascade - the host socket is closed, which triggers dispatcher to exit, and when its sockets are reaped, this then causes the daemons to exit. If just one 'detritus' then how did the daemons exit?

I'm not entirely sure what this is, but I'm guessing some artefact of R CMD check on Windows. Probably not much help here, but a general debugging tip is to specify output = TRUE in the call to daemons(). Then if there are any errors they are output to the console.

[HenrikBengtsson]

Thanks. I've tried with

library(mirai)

message("Launch two workers")
mirai::daemons(2, output = TRUE)
print(mirai::daemons())

message("Reset")
mirai::daemons(0)
print(mirai::daemons())

Sys.sleep(5)

message("Reset")
mirai::daemons(NULL)
print(mirai::daemons())

Sys.sleep(5)

message("Reset")
mirai::daemons(0)
print(mirai::daemons())

Sys.sleep(5)

Unfortunately, it provides no clues. Here's the tests/mirai.Rout file:

R version 4.3.3 (2024-02-29 ucrt) -- "Angel Food Cake"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(mirai)
> 
> message("Launch two workers")
Launch two workers
> mirai::daemons(2, output = TRUE)
[1] 2
> print(mirai::daemons())
$connections
[1] 1

$daemons
                               i online instance assigned complete
ipc://2256950364ce40045a41b00c 1      0        0        0        0
ipc://f4eeb403366739704c1109bf 2      0        0        0        0

> 
> message("Reset")
Reset
> mirai::daemons(0)
[1] 0
> print(mirai::daemons())
$connections
[1] 0

$daemons
[1] 0

> 
> Sys.sleep(5)
> 
> message("Reset")
Reset
> mirai::daemons(NULL)
[1] 0
> print(mirai::daemons())
$connections
[1] 0

$daemons
[1] 0

> 
> Sys.sleep(5)
> 
> message("Reset")
Reset
> mirai::daemons(0)
[1] 0
> print(mirai::daemons())
$connections
[1] 0

$daemons
[1] 0

> 
> Sys.sleep(5)
> 
> 
> 
> proc.time()
   user  system elapsed 
   0.14    0.04   15.60 

[shikokuchuo]

Right, if no errors then shouldn't be a more general issue. I see above that at the time the status is printed, the daemons have not yet connected. What happens if you call daemons(2, local_url()) instead? This just sets up dispatcher without actually launching any daemons. If this is fine then it is likely just some complicated timing issue that one of the processes is reaped too quickly by R CMD check.

[HenrikBengtsson]

I'm not entirely sure what this is, but I'm guessing some artefact of R CMD check on Windows.

Yes, these type of Rscript<hash> files/folders may survive on MS Windows, because when launching child processes via system(), Windows does (always?) not terminate them when the parent terminates. On Unix-like systems, that is taken care of automagically by the OS, so it's rare to see this there. We can see other types of detritus files, but those are more temporary side-effect files that the package maintainer should clean out.

FWIW, I've done lots of troubleshooting in the past of this, and it's always been MS Windows background processes that do not terminate. I could consistently reproduce it with Futureverse, parallelly, and regular parallel clusters. I also thought there was something broken with R CMD check, but after endless trial'n'error troubleshooting, I've always found that they are not false positives. So, I'd bet on that horse here too, i.e. there's a mirai background R process still running, or abruptly terminating so it doesn't clean up after itself (unlikely). I've summarized my previous findings in https://parallelly.futureverse.org/reference/makeClusterPSOCK.html.

I also reported this to BugZilla (https://bugs.r-project.org/show_bug.cgi?id=18133). Now, re-reading that thread, reminded me that this is particularly a problem when a child process is launching another Rscript process (see my comment in the thread). So, is mirai launching multiple-levels of nested Rscript processes here? That could be useful to known in trying to track this down.

[shikokuchuo]

That bugreport is certainly an interesting read! Yes, dispatcher is a child process which then launches grandchildren. But your guess as to the cause doesn't apply here - as I mentioned above, the order of exit can only be a cascade. So perhaps it is something else going on specific to Windows.

EDIT: actually re-reading, this might be applicable - I'll have to look into it more closely. Yes, not quite the same, for starters, the tempdirs are definitely not left behind on Linux.

[HenrikBengtsson]

Right, if no errors then shouldn't be a more general issue. I see above that at the time the status is printed, the daemons have not yet connected. What happens if you call daemons(2, local_url()) instead? This just sets up dispatcher without actually launching any daemons. If this is fine then it is likely just some complicated timing issue that one of the processes is reaped too quickly by R CMD check.

Just tried, and, yes, the "detritus in the temp directory" NOTE is no longer there on MS Windows, cf.
https://github.com/HenrikBengtsson/teeny/actions/runs/8620412600.

On a side note, and unrelated to this topic, this resulted in the following macOS (only) error:

  > mirai::daemons(2, url = local_url(), output = TRUE)
  Error in listen(nsock, url = nurl, tls = tls, error = TRUE) : 
    12 | Entry not found
  Calls: <Anonymous> -> listen
  Execution halted
  Error in init_monitor(sockc = sockc, envir = envir) : 
    initial sync with dispatcher timed out after 10s
  Calls: <Anonymous> -> init_monitor

That could be specific to GitHub Actions.

[HenrikBengtsson]

@shikokuchuo @wlandau-lilly , are either of you using daemons(..., dispatcher = TRUE) in unit tests of your packages? I cannot seem to find a way to work around this other than skipping examples/tests on MS Windows.

[wlandau]

I noticed this in crew a couple versions ago, and it was tough to track down. IIRC it happens in daemons() when I don't specify a URL. In my case, I almost always want to set the URL, so it's not a big deal to skip those test and I haven't investigated further than that.

[shikokuchuo]

I see that there is now a patch on Bugzilla for your bug report, hence it might not be an issue for much longer. Having said that, I don't actually recall this ever being a specific issue on Windows. But it's been a while since I've specifically looked at this platform so I will need to check and get back to you.

[shikokuchuo]

I moved a dispatcher test to run on Windows and it is not showing the detritus issue. On Github, R-universe or indeed Win-builder. That test does happen to use daemons(NULL) though.

daemons(NULL) sends an exit signal to the daemons, so those should now exit first, before the host and dispatcher processes are torn down.

[HenrikBengtsson]

Thanks. I thought I tried that too among a lot of other trial'n'error attempts, but I only did so with daemons(0) first and only then daemons(NULL). I can confirm that starting out with daemons(NULL) avoids them problem. Here's the test results, with code and lots of verbose output for future references;

R Under development (unstable) (2024-04-15 r86425 ucrt) -- "Unsuffered Consequences"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(mirai)
> 
> tmp_before <- dir(dirname(tempdir()), full.names = FALSE)
> 
> message("Launch two workers")
Launch two workers
> mirai::daemons(2, output = TRUE)
[1] 2
> str(mirai::daemons())
List of 2
 $ connections: int 1
 $ daemons    : int [1:2, 1:5] 1 2 0 0 0 0 0 0 0 0
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:2] "ipc://0501ff76a21b59454f288c30" "ipc://0ce59d18945d7d8e674abb6b"
  .. ..$ : chr [1:5] "i" "online" "instance" "assigned" ...
> 
> main <- data.frame(name = "main", pid = Sys.getpid(), tempdir = basename(tempdir()))
> ms <- lapply(1:2, FUN = function(kk) {
+   mirai({
+     Sys.sleep(1)
+     data.frame(name = "worker", pid = Sys.getpid(), tempdir = basename(tempdir()))
+   })
+ })
> workers <- lapply(ms, FUN = function(m) call_mirai(m)$data)
> workers <- do.call(rbind, workers)
> 
> message("Added temporary files and folders:")
Added temporary files and folders:
> tmp_after <- dir(dirname(tempdir()), full.names = FALSE)
> tmp_diff <- setdiff(tmp_after, tmp_before)
> print(tmp_diff)
[1] "Rscript17eca8f20" "Rscriptc98a8fcb"  "Rscriptf88a8fcb"  "RtmpMpRj8O"      
[5] "RtmpaIBGPZ"       "RtmpiO2B71"      
> 
> res <- rbind(main, workers)
> res$pid_hex <- sprintf("%x", res$pid)
> res$detritus <- NA_character_
> for (kk in 1:nrow(res)) {
+   pattern <- sprintf("^Rscript%s", res$pid_hex[kk])
+   match <- grep(pattern, basename(tmp_diff), value = TRUE)
+   if (length(match) > 0) res$detritus[kk] <- match
+ }
> message("Detritus information:")
Detritus information:
> print(res)
    name  pid    tempdir pid_hex        detritus
1   main 5980 RtmpuWeJ9F    175c            <NA>
2 worker 3976 RtmpiO2B71     f88 Rscriptf88a8fcb
3 worker 3224 RtmpaIBGPZ     c98 Rscriptc98a8fcb
> 
> message("Reset")
Reset
> mirai::daemons(NULL)
Error in (function (aio)  : attempt to bind a variable to R_UnboundValue
Calls: <Anonymous> -> <Anonymous> -> .Call
Execution halted
Error in (function (aio)  : attempt to bind a variable to R_UnboundValue
Calls: <Anonymous> -> <Anonymous> -> .Call
Execution halted
[1] 0
> str(mirai::daemons())
List of 2
 $ connections: int 0
 $ daemons    : int 0
> 
> message("Added temporary files and folders (+5s):")
Added temporary files and folders (+5s):
> tmp_after <- dir(dirname(tempdir()), full.names = FALSE)
> tmp_diff <- setdiff(tmp_after, tmp_before)
> print(tmp_diff)
character(0)
> stopifnot(length(tmp_diff) == 0)
> 
> 
> proc.time()
   user  system elapsed 
   0.17    0.04    1.62 

[wlandau]

crew has been using daemons(n = 0L) thus far. Would it be reasonable to switch to daemons(n = NULL) for both local and remote daemons? Anything that could go wrong? It seems to work fine when I try it.

From the documentation of mirai::daemons():

To reset persistent daemons started with autoexit = FALSE, use daemons(NULL) instead, which also sends exit instructions to all connected daemons prior to resetting.

From my perspective, this can only help crew.

[shikokuchuo]

@wlandau I only recommend use of daemons(NULL) for the documented case you mentioned above. Rest assured, if I think there is something here that affects crew I will be sure to let you know.

The way I see it, this is just a possible way of appeasing R CMD check on Windows (which is a bug as proposed by Henrik on R bugzilla, where there is already a proposed patch).

[HenrikBengtsson]

Clarification question: What's the difference between daemons(NULL) and daemons(0) under the hood? For instance, does daemons(0) try to shut down things nicely, whereas daemons(NULL) just kills things (cf. "Error in (function (aio) : attempt to bind a variable to R_UnboundValue")?

[HenrikBengtsson]

BTW, I'm not 100% sure I agree with it being a "bug" as in "which is a bug as proposed by Henrik on R bugzilla, where there is already a proposed patch". I see that's more as a feature request to ignore more stray files on MS Windows analogously to how some other files are already ignored. Not sure what the history of ignoring such files in the first place comes from - maybe they couldn't figure out how to shut everything down nicely. Ideally, all sub-processes should shut down nicely and always clean up after themselves. OTH, I can see how that could become too tedious to get correct everywhere, and it's simply just easier to fall back on the operating system to eventually cleanup our TMPDIR at a later stage (possibly much later or only when user manually requests it). So, if I understood the problem here, ideally daemons(0) would result in the daemons first shutting down and then, and only then, the dispatcher process will shut down.

[shikokuchuo]

Daemons exit their loop automatically whenever their connection is broken (through synchronisation primitives), and exit gracefully. This is the intended behaviour.

It is possible however to launch persistent daemons, in which case this mechanism is disabled, this means that an exit signal needs to be sent which causes the daemons to exit via an evaluation error.