Skip to content

Python script to download data from the Image SFTP server

Notifications You must be signed in to change notification settings

imagen2/imagen_sftp_downloader

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

sftp_downloader

This code may help to download IMAGEN data via sftp. You need to....

  • ...have login credentials and enter these into the file secret.py (use secret_template.py to create it.) Caution: never push the file with your login credentials to a public github repository
  • ...know what you want to download, look at the folder structure using an sftp browser before using this script

Prerequisites

  • Python 3 (tested with 3.7+)
  • pysftp

How-To

Folder structure

The downloader assumes the following folder structure on the sftp server:

remote_path/{time_dirs}/{overall_dirs}
or
remote_path/{time_dirs}/intermed_dir/{subjs}/{dirs}/{subj_files}

Download modes

There are four modes that help to do different things

mode = "dirs"   # "files" or "dirs" or "subjects" or "overall"
  1. "overall" mode: download a set of given folders recursively (including all subdirectories and files)
  2. "subjects" mode: download given subject folders recursively (including all subdirectories and files)
  3. "dirs" mode: download specific subdirectories within subject folders recursively
  4. "files" mode: download files which match specific patterns within specific subdirectories within subject folders

Steps

  • clone the repository
  • use secret_template.py to create a new file secret.py
  • enter your login information in secret.py
  • use settings_template.py to create a new file settings.py
  • enter your local path settings and your download definitions in settings.py (some examples are given in settings_template.py)
  • start the script:
python get_data.py
  • logfiles are created for basic information (info_logger*) and debugging information (debug_logger*) to check if something did not work.

Limitations

  • pysftp may not work properly with windows (e.g. recursive downloads may be buggy)
  • download is not super fast (own experience: 1.5GB per hour)

Caution

  • not extensively tested for all use-cases, use at your own risk.

About

Python script to download data from the Image SFTP server

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages