php symfony digitalobject:regen-derivatives - missbehaving

[mtilnets]

Current Behavior

get onto cli in the atom path and execute php symfony digitalobject:regen-derivatives -o -n
the derivates will be created and populated to atom, but they remain afterwards also in the temp-folder (in my case in /tmp/)

occassionaly also sfWebBrowser in the script, mostly the sfCurlAdapter seams to run into an OOM-Error, this happens in case of downloading multiple large files, in my case video-files from a local web-server which can have a size of multiple GB.
And additionaly the download rate drops to arround 2MB/s after it begins to download the second file. Normaly the link delivers up to 100MB/s on the first file after starting the script it maintains this speed.

The method to download files used in the digitalobject procedures are only suitable for relative small elements like digital images or PDF, but not for A/V media.

Expected Behavior

the temporary files shall be removed directly after being copied to their destination folder.

Downloading the files should not happen as bistream to memory (like now with sfWebBrowser to $contents in qbitDigitalObject.php in the downloadExternalObject procedure) but it should happen as stream directly to the temporary file, so the other procedures should use these temp-file instead the $contents variable

Possible Solution

for case 1:
in /lib/model/qubitDigitalObject.php in createAudioDerivative() and createVideoDerivative() finaly the temporary file has to be unlinked.
in both procedures this unlink does not exist.
in both cases only unlink($derivativeFullPath); ist to ad right behind $derivative->save($connection);

for case 2:

in downloadExternalObject()
the download should not be done into a variable in the shared memory (what causes memory exhaustion, 100% cpu load
and a massive drop in download-speed after )
but chunk-wise into a temporary file, like it is also build afterwards in saveAndAttachFileContent()

A possible workaround for consuming ram in the multiple size of the loaded file is to unset the $contents or $browser variables right after there were processed and force garbage-collect with gc_collect_cycles();
The exceeding of the RAM for the sript with large files and the issue, that the second download an any further are limited to around 1MB/s have something to do with the sfCurladapter user in sfWebBrowser, there or in the cURL Module in PHP7.4 seemst to be a problem. A quick resolve to the problem is to swicht to sfFopenAdapter. then memory frees completely up with the forced garbage collect and the download performace maintain stable at high rates in every iteration

Context and Notes

No response

Version used

AtoM 2.8.0 -193

Operating System and version

Ubuntu 20.04

Default installation culture

en

PHP version

No response

Contact details

[email protected]