Support non-circular skyregions

[timstaley]

This could be quite tricky, but is crucial to stand a chance of supporting optical and/or mosaic image data. (cf https://redmine.astron.nl/lofar_issuetracker/issues/6577).

Possible options include basic support for rectangular regions (cf optical CCDs) and/or composite regions composed of a bounding circle and sub-circles (mosaics).

Depending on whether or not we still support multiple DBMS back-ends by this point, we may be able to leverage an off-the shelf solution e.g. PostGIS

[AntoniaR]

So I agree that we need to do this in future, but completely disagree that this should be high priority as there are a few issues with dealing with this type of image. Specifically, the noise properties in the mosaics change by so many orders of magnitude that it will be almost impossible to pull out the true transient sources. Until we can subdivide images into smaller regions, there is no point in supporting non-circular sky regions.

Personally, I would say that it is far easier to quickly slice up the mosaic images into rectangles that can be then input into TraP which can get around the other issues while we look at other things. Or even better - don't mosaic until this is all ready. So basically, as there are a few ways people can easily work round this issue and a number of issues that need resolving before this issue, this issue should be a low priority.

[jbroderick1]

Don't necessarily agree with 'many orders of magnitude' for LOFAR mosaics. Just my opinion, but I'd like to see this have reasonably high priority still.

[timstaley]

Righto. I'll leave it as-is for now - it's targeted at milestone 4.0, so we can re-evaluate when we hit 3.0 (end of this year, earliest).

[AntoniaR]

So because of how PySE measures the RMS map and we take the max and min from that map, for a single image (not mosaicked) from the RSM dataset the RMS min is in the range [0.009, 0.038] Jy and the RMS max is in the range [0.027, 4.3] Jy with typically 1 order of magnitude between the two values. Note this is also only for the inner part of the images and not the outer regions where the noise gets higher and that these numbers are only for images which passed a strict noise threshold.

Taking 6 beams at the same frequency, the same observation and only using the inner region of the image (e.g. run6_L99999_PT39_SAP00*_BAND01_FINAL.MS.dppp.img.restored.corr and inside the primary beam FWHM), the min RMS is 0.017 and the max RMS is 0.347. So my estimate that this would be equivalent to 1 image with this range - although this is easily testable by putting a few mosaics into TraP. Note, I randomly chose a good observation where all the images passed a strict image RMS quality control setting - so I would say this is probably a "best case" and not surprising that it is only just over 1 order of magnitude.

The biggest problem is when one or more of the beams is bad - e.g. due to bad calibration. Without mosaicking, this beam would be easily rejected by the noise threshold. In a mosaic that beam would not only still remain in the dataset but would also push the max RMS noise up by orders of magnitude. Another case to think about is when the Galactic plane is across a beam or two - this also raises the max noise and, hence, you would be a lot less sensitive to transients in other regions of the image.

Also, what are you going to do if the restoring beam changes shape between the different beams? This might be difficult to obtain with LOFAR mosaics but is a major issue for data that I am working with from ATCA.

So basically, I completely agree that TraP should be able to deal with any shape images but we need to be able to subdivide images first so that:

• quality control tests are conducted on regions not the full image
• transient search is using much smaller parts of the image for the RMS min/max values
• different restoring beams for different images can be dealt with
• consider implementing the sourcefinder parallelisation code so that we can more quickly process large images

I would be very happy to see all these issues form Release 4.0.

[jbroderick1]

I think that is a very good plan of attack.

At the moment, the different restoring beams is dealt with by taking the average and doing some sort of convolution (I think?....actually I should check this in George's script). Ideally you would convolve everything to the same resolution before doing any sort of mosaicking.

Btw for ATCA, you can do this by using the 'fwhm' parameter in 'invert' in Miriad. Or manually setting the restoring beam in 'restor' if you don't really care about extended emission.

[AntoniaR]

This is an ongoing wish list item but not essential for R7. Moving to the long term milestone.