better treatment of evicted batch workflow pods

[tiborsimko]

A workflow is reported as "running":

reana=> select created,updated,status from Workflow where id_='ef2dd704-2924-47eb-98b6-9a3d3394f9e5';
         created          |          updated           | status  
--------------------------+----------------------------+---------
 2021-01-19 15:06:41.7818 | 2021-01-19 15:37:46.201782 | running

but it was actually evicted:

$ kubectl get pods | grep -i evicted 
reana-run-batch-ef2dd704-2924-47eb-98b6-9a3d3394f9e5-9wp6h        0/2     Evicted     0          2d3h

due to:

$ kubectl describe pod reana-run-batch-ef2dd704-2924-47eb-98b6-9a3d3394f9e5-9wp6h 
...
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: ephemeral-storage. 

Current status: the workflow controller does not seem to notice, so the workflow is reported as still running to the user.

Expected behaviour: it would be useful if REANA platform notices these "evicted" situations and informs user properly back.

[tiborsimko]

Update: seen similar cases where the workflow was reported as "failed", the run-batch pod was "Evicted", but the task remained actively visible in kubectl get jobs. So the above happens not only in the "running" context, but also when a workflow is recognised as "failed" due to eviction due to lows storage on node, the pods are not garbage-collected still.