Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to set ISCE stackSentinel.py parameters in order to run MiaplPy? #60

Open
leeyhgeo opened this issue Aug 14, 2023 · 21 comments
Open

Comments

@leeyhgeo
Copy link

Hello,

I used to analyze Sentinel1 images with MintPy and I found that MiaplPy seems to need different input. However, I'm confused about how to set stackSentinel.py parameters so that I can get coregistered SLC data in full resolution. Is there any document/discussion that could be referenced? (I can hardly find one so far)

Thank you

@mirzaees
Copy link
Collaborator

Hi @geoglee
MiaplPy can read data from 'interferogram' and 'slc' workflow of stackSentinel.py

If your current directory is the project directory, then setting the autoPath to yes would read all data automatically.
You might need to rename your template to reflect your project, for example if your project directory name is 'testProject' then rename your template to testProject.template:

./testProject/testProject.template

miaplpy.load.autoPath = auto # [yes, no] auto for no

Also you can manually set path to data as bellow in your template file:

miaplpy.load.metaFile = ${PROJECT_DIR}/reference/IW*.xml
miaplpy.load.baselineDir = ${PROJECT_DIR}/baselines
miaplpy.load.slcFile = ${PROJECT_DIR}/merged/SLC/*/*.slc.full
miaplpy.load.unwFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.unw
miaplpy.load.corFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.cor
miaplpy.load.connCompFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/*.unw.conncomp
miaplpy.load.ionoFile = None
miaplpy.load.intFile = None
miaplpy.load.demFile = ${PROJECT_DIR}/merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile = ${PROJECT_DIR}/merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile = ${PROJECT_DIR}/merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile = ${PROJECT_DIR}/merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile = ${PROJECT_DIR}/merged/geom_reference/waterMask.rdr.full
miaplpy.load.bperpFile = None

@leeyhgeo
Copy link
Author

Thank you so much @mirzaees !
I've tried your suggestion but I ran into some path problems. I hope I could solve it soon.
At the same time, I'm using stackStripMap.py to process ERS and Envisat data.
Is it possible as well to run products from stackStripMap.py in MiaplPy?

@mirzaees
Copy link
Collaborator

you are very welcome @geoglee
I may be able to help you if you send the error you get and your template

Regarding stripmap, yes, I have run it for TerraSAR data before and it worked
however I have never tested ERS or Envisat, let me know if you encounter a problem

@Jesse-kearse
Copy link

Hello

I have been using MintPy and MiaplPy, and have managed to get good results. Recently, I have had an issue when loading an SLC stack.

The SLC stack was generated with the following parameters:
stackSentinel.py -s /insar/raw/track_08/ -d /insar/raw/dem/dem.wgs84 -b '-38.16 -37.53 176.07 177.73' -n '1 2' -m 20180112 -a ./aux/ -o ./orbits -C NESD -W slc

I ran the MiaplPy software with the suggested command "miaplpyApp.py kawerau.cfg --runfiles --dir ."

the prepc_slc_isce.py application crashes, and I am unsure why. I thought it may be due to

  1. "miaplpy.subset.lalo" setting, so I tried with various extents and with no subset, and it was the same result
  2. "miaplpy.load.startDate" or "maiplpy.load.endDate" so I tried with and without dates and the result was the same.

The only difference I could think of between this SLC stack and other SLC stack that worked fine with miaplpy, is co-registration method: "NESD" was used here, and I had used "geometry" in the past with the SLC stack that were loaded into Miaplpy without issue. Perhaps the the choice of number of swaths could be a factor? I am stuck as to the issue.

Below is the report of the run, is there anything else I could provide to help diagnosing the issue?

Thanks for any help

--RUN-at-2023-09-25 11:35:24.796217--
Current directory: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy
Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Project name: kawerau

20230925:113524 * miaplpyApp.py kawerau.cfg --runfiles --dir .
Project name: kawerau
Go to work directory: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy
copy default template file /home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/kawerau.cfg
update default template based on input custom template
No new option value found, skip updating /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/smallbaselineApp.cfg
copy kawerau.cfg to inputs directory for backup.
copy smallbaselineApp.cfg to inputs directory for backup.
copy kawerau.cfg to pic directory for backup.
copy smallbaselineApp.cfg to pic directory for backup.
read default template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/smallbaselineApp.cfg
copy default template file /home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/objects/../defaults/miaplpyApp.cfg to work directory
update default template based on input custom template
miaplpy.multiprocessing.numProcessor: auto --> 40
miaplpy.load.processor: auto --> isce
miaplpy.load.autoPath: auto --> yes
miaplpy.load.slcFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/SLC//.slc.full
miaplpy.load.startDate: auto --> 20180112
miaplpy.load.endDate: auto --> 20200114
miaplpy.load.metaFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW*.xml
miaplpy.load.baselineDir: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/baselines
miaplpy.load.demFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile: auto --> /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile: auto --> None
miaplpy.load.unwFile: auto --> ./inverted/interferograms_single_reference//fine.unw
miaplpy.load.corFile: auto --> ./inverted/interferograms_single_reference/
/fine.cor
miaplpy.load.connCompFile: auto --> ./inverted/interferograms_single_reference//.unw.conncomp
miaplpy.subset.lalo: auto --> -38.117:-37.995,176.651:176.913
mintpy.load.processor: auto --> isce
copy miaplpyApp.cfg to inputs directory for backup.
read default template file: /Volumes/GeoPhysics_28/users-data/kearsejess/insar/work/track_175/miaplpy/miaplpyApp.cfg
SAR platform/sensor : None
processor: isce
check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products
prep_slc_isce.py -s /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/SLC -f .slc.full -m /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW1.xml -b /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/baselines -g /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/merged/geom_reference --force
extract metadata from ISCE/topsStack xml file: /Volumes/GeoPhysics_24/users-data/kearsejess/insar/work/track_08/slc_2015_2021/reference/IW1.xml
extract metadata from geometry files: ['hgt.rdr.full', 'lat.rdr.full', 'lon.rdr.full', 'los.rdr.full']
extract metadata from geometry files: ['hgt.rdr', 'lat.rdr', 'lon.rdr', 'los.rdr']
Traceback (most recent call last):
File "/home/kearsejess/conda/envs/mintpy_new/bin/prep_slc_isce.py", line 8, in
sys.exit(main())
^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/prep_slc_isce.py", line 362, in main
metadata = extract_isce_metadata(inps.metaFile,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/miaplpy/prep_slc_isce.py", line 145, in extract_isce_metadata
metadata_multi_looked = isce_utils.extract_geometry_metadata(geom_dir, metadata_multi_looked)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/isce_utils.py", line 559, in extract_geometry_metadata
data = readfile.read(geom_file, box=box)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 310, in read
data, atr = read_binary_file(fname, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 679, in read_binary_file
data = read_binary(
^^^^^^^^^^^^
File "/home/kearsejess/conda/envs/mintpy_new/lib/python3.11/site-packages/mintpy/utils/readfile.py", line 1914, in read_binary
data = np.fromfile(fname, dtype=data_type, count=count).reshape(-1, width
num_band)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: cannot reshape array of size 15521796 into shape (43903)

@Jesse-kearse
Copy link

P.S. I am running the latest version of MiaplPy V0.2.1, which was downloaded ~20092023

@mirzaees
Copy link
Collaborator

mirzaees commented Sep 26, 2023

Hi @Jesse-kearse
Looks like your program crashes when reading geometry files. One thing that comes to my mind is the *.rsc files, If you have run mintpy before this, it has created *.rsc metadata in the geometry directory which are associated with multilooked files. now you are reading the full res and miaplpy tries to create *.rsc for full res but it first checks whether they exist in the directory. So please delete them (all *.rsc files in merged/geom_reference directory and merged/SLC/dates/* directory) and try again. let me know if that solves the issue

@Jesse-kearse
Copy link

Hi @mirzaees , yes I had run mintpy on the same SLC stack prior to miaplpy. Thanks for your advice! I will try your suggestion and let you know how it goes.

Cheers

@Jesse-kearse
Copy link

hi @mirzaees , Actually, I had not run mintpy on this particular SLC stack, and there were no .rsc files in the /merged/geom_reference/ dir or the /merged/SLC/*/ directory. Any other ideas?

Thanks again for your help.

@mirzaees
Copy link
Collaborator

Hi @Jesse-kearse,
I need a subset from your data as a test to run it and see if I can reproduce your error. Can you provide me your merged folder with geom-reference, and 2 dates in SLC and reference+baseline folders in a zipfile somewhere?

@gjustin40
Copy link

I also have a similar issue. @mirzaees
I created a stack using ISCE2, and ultimately, the problem is that the slc.full data is missing.

miaplpy.load.slcFile        = ../merged/SLC/*/*.slc.full  #[path2slc_file]

After running ISCE2 command, the slc.full file was not generated.
The ISCE2 command I executed is as follows:

dem.py -a stitch -b {bbox lat/lon} -r -s 1 -c
stackSentinel.py -s SLC/ -d DEM/*.dem.wgs84 -a AuxDir/ -o Orbits/ -b {bbox lat/lon} -c 2
# run all run_files

I successfully used MintPy with the files generated by above command.

I know there are a lot of example of command to make run_files in ISCE,
What command should I execute in ISCE2 to generate slc.full files?

Thanks

@Jesse-kearse
Copy link

Hi @mirzaees

Thank you for offering to take a look.

I have created the a zip archive "SLC_zip.zip" that contains the files you requested. You can download with the following command:

rsync -av --progress rsync://[email protected]/kearsejess/

password: gJbmvnmtUU

Thanks again.

@mirzaees
Copy link
Collaborator

Hi @Jesse-kearse
Thanks for providing your data. which workflow of stackSentinel.py have you used? if that is interferogram or slc, then it should work the way it is without any touching but from your data I realize you may have done some separate/post processing or the merging step was not complete. that is because your multilooked version of geometry files are not compatible with their shape in VRT. Also if you display them with view.py of mintpy, you get a similar error. Anyways I removed that part of the code that relies on multilooked version and should not be a problem any more, just reinstall with:

cd MiaplPy; git pull
python -m pip install .

one more issue is about the baselines you provided which seems to be a raster file for each date. however, miaplpy assumes text files in the following format if your reference date is 20180112:

20180112_20180301/20180112_20180301.txt
20180112_20180313/20180112_20180313.txt

I don't know why your baselines are like this but stackSentinel.py should create those text files. You will still get error if your baselines are not in that format

@mirzaees
Copy link
Collaborator

Hi @gjustin40
As long as the full res VRT files exist and you have not deleted the other folders in the project, you are fine and it should be able to read them. However, as I mentioned above, if you have run mintpy before miaplpy, clean all .rsc metadata and run miaplpy again

@Jesse-kearse
Copy link

Thanks for taking the time to look at the files, @mirzaees , I really appreciate your help.

The workflow I used was SLC: "stackSentinel.py -s /insar/raw/track_08/ -d /insar/raw/dem/dem.wgs84 -b '-38.16 -37.53 176.07 177.73' -n '1 2' -m 20180112 -a ./aux/ -o ./orbits -C NESD -W slc"

I realise now that the post processing that I had done with StaMPS (which includes a multi-looking step) was probably the reason for the modified geometry files. But with your recent update this will no longer be an issue! thanks, I will git pull and re-install.

The baseline files I sent were from ./merged/baselines/ dir. Sorry I misinterpreted which baseline information you wanted. The files in ./baselines/ dir are in the correct format: 20180112_20180301/20180112_20180301.txt, so that should be fine.

I will let you know how it goes.

Thanks again.

@Jesse-kearse
Copy link

Following git pull and re-installation it was a successful completion of "miaplpyApp file.cfg --runfiles --dir ."

Thanks @mirzaees

@mirzaees
Copy link
Collaborator

You are very welcome @Jesse-kearse
I will close this issue

@leeyhgeo
Copy link
Author

leeyhgeo commented Oct 24, 2023

Hi @geoglee MiaplPy can read data from 'interferogram' and 'slc' workflow of stackSentinel.py

If your current directory is the project directory, then setting the autoPath to yes would read all data automatically. You might need to rename your template to reflect your project, for example if your project directory name is 'testProject' then rename your template to testProject.template:

./testProject/testProject.template

miaplpy.load.autoPath = auto # [yes, no] auto for no

Also you can manually set path to data as bellow in your template file:

miaplpy.load.metaFile = ${PROJECT_DIR}/reference/IW*.xml
miaplpy.load.baselineDir = ${PROJECT_DIR}/baselines
miaplpy.load.slcFile = ${PROJECT_DIR}/merged/SLC/*/*.slc.full
miaplpy.load.unwFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.unw
miaplpy.load.corFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/fi*.cor
miaplpy.load.connCompFile = ${WORK_DIR}/inverted/interferograms_${int_type}/*/*.unw.conncomp
miaplpy.load.ionoFile = None
miaplpy.load.intFile = None
miaplpy.load.demFile = ${PROJECT_DIR}/merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile = ${PROJECT_DIR}/merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile = ${PROJECT_DIR}/merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile = ${PROJECT_DIR}/merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile = ${PROJECT_DIR}/merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile = ${PROJECT_DIR}/merged/geom_reference/waterMask.rdr.full
miaplpy.load.bperpFile = None

@mirzaees Sorry for the really late reply.

I run my stackSentinel.py with the following parameter stackSentinel.py -s ~/slc -o ~/orbits -a ~/S1_auxiliary_files -d ~/dem/dem.wgs84 -c 3 -z 3 -r 15 -b '22.3 24.4 119.9 120.6' and this is how it looks like under the folder ${PROJECT_DIR}/merged/SLC/*
截圖 2023-10-25 上午1 21 08
I cannot find any files end with *.slc.full, and so as files under the geom_reference folder as below
截圖 2023-10-25 上午1 24 46

I wonder if the problem arises from my multilooking settings of -z 3 -r 15 above, or maybe other issues.

Thanks!

@mirzaees mirzaees reopened this Oct 26, 2023
@mirzaees
Copy link
Collaborator

@geoglee
I don't see any problem with your isce outputs,
as long as the vrt files exist in your folder, miaplpy will read the data.
you need to set up your template similar to the example above.
I need to see your error to be able to help you, can you run and send me your error message?

@Jesse-kearse
Copy link

@geoglee

If you only have the "merged" directory and sub directories, you may be missing the files that the .slc.full.vrt files are referencing. In my workflow, .slc.full.vrt files point to burst.slc.vrt files in the "coreg_secondarys" directory, one above "merged". The raw data is contained in the burst.slc files within the same directory. I ran into this issue by mistake when I deleted a whole bunch of files (to gain extra disc space) thinking I only needed the merged// files. You can check by opening the *.slc.full.vrt files and checking the path in

@Jesse-kearse
Copy link

...checking the path in SourceFilename relativeToVRT

@leeyhgeo
Copy link
Author

@mirzaees @Jesse-kearse

Thank you a lot for helping me! For the path problem, awkwardly, I rechecked the template and just found that I didn't turn on the auto path function since I thought the default setting (auto) was yes till I noticed the comment behind miaplpy.load.autoPath. As I turn on the auto path function, Miaplpy seems to work right now (though it's still running). Hope it can operate normally till the final step. I'll report the outcome later.

Another question about the stackSentinel.py settings. I'm not sure whether multilooking (-z, -r) and num_connections (-c) will affect the Miaplpy processing results. Any ideas?

Besides, I already put the keyword "Sen" into the file name of my template, but the message still outputs SAR platform/sensor : None. How to set it properly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants