TraP slowdown when image store is enabled

[gijzelaerr]

Alexander his team indicated that TraP 4.0rc1 becomes quite slow when image store is enabled. When disabled TraP is super fast. This is because the images are stored in the database in serial (that is how SQL works) and not in parallel anymore (MongoDB was filesystem based). So question now is, is the slowdown too much or not. @AntoniaR indicated she doesn't need the image store, so for her this shouldn't be a problem.

If the slowdown is too much, we have various options:

• Come up with a new workflow where we don't copy the images to a database, but somehow still make it possible banana can access the images that have been processed. In theory this is possible, it is just more of a business logic problem, banana needs to run on the same system as where the images are or needs to have access to the images in a similar way.
• Switch back to the old method, store everything in MongoDB. I think this is a bad idea, since we have no proper way of managing the images. Last time vlo was full @Error323 just threw away all the data since there was no easy way to determine which images belong to which dataset.
• Switch to a hybrid method where we store a more complex key in MongoDB so we can determine to which dataset / database an image belongs.

I prefer to leave it just the way it is, since it makes it much easier to manage the data and it requires less services so simplifies TraP.

[AntoniaR]

Clarification, I said that I stored images in other ways. Some kind of intelligent image store will be needed for multiple datasets, not just AARTFAAC, and I had basically proposed your first option when we discussed this - i.e. an image store separate to TraP but that Banana accesses.

So I think working round it for now is ok but we will need to quantify the slowdown and work round it better in the longer term.

[gijzelaerr]

point one is a logistics issue, not a software issue. I think how that turns out the AARTFAAC team has to decide when things get more a final form. I propose we just leave it for now as is, storing images this way is for debugging or experimenting.

[AntoniaR]

Image storage is not scalable for the long term use of the TraP. This option should be completely removed for R7.