Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable PDS prepare to upload to shared storage #3026

Closed
lorriborri opened this issue Mar 18, 2024 · 0 comments · Fixed by #3131
Closed

Enable PDS prepare to upload to shared storage #3026

lorriborri opened this issue Mar 18, 2024 · 0 comments · Fixed by #3131
Assignees
@lorriborri
Labels
prepare
Milestone
Prepare Wrapper 1...

Comments

@lorriborri
Copy link
Member

Situation

The PDS prepare should be able to load prepared data into the shared storage so that it is available for further scans.

Wanted

When data has been successfully prepared, the PDS prepare should upload it to the shared storage.

Solution

Introduce environment variable PDS_PREPARED_FOLDER, the PDS should upload all data in this folder to the storage
@lorriborri lorriborri assigned lorriborri and unassigned de-jcup May 6, 2024
lorriborri added a commit that referenced this issue May 14, 2024
@lorriborri
Added upload classes and archive support for prepare #3026
c98952d
@lorriborri lorriborri mentioned this issue May 15, 2024
Merged
4 tasks
lorriborri added a commit that referenced this issue May 15, 2024
@lorriborri
moved git download into folder #3026
6e5f747
lorriborri added a commit that referenced this issue May 21, 2024
@lorriborri
Added test for UploadService #3026
74f6c0c
lorriborri added a commit that referenced this issue May 22, 2024
@lorriborri
Added unit tests for upload #3026
00a4ff3
lorriborri added a commit that referenced this issue May 24, 2024
@lorriborri
Refactored PDSStorageConstants #3026
e8ea8e6
lorriborri added a commit that referenced this issue May 24, 2024
@lorriborri
Added change requests #3026
9407d38
lorriborri added a commit that referenced this issue May 24, 2024
@lorriborri
Added change requests #3026
311dbee
lorriborri added a commit that referenced this issue May 24, 2024
@lorriborri
Added change requests #3026
43a6c11
lorriborri added a commit that referenced this issue May 27, 2024
@lorriborri
Added change requests #3026
d2b7d1c
lorriborri added a commit that referenced this issue May 27, 2024
@lorriborri
Moved LogSanitizer in pds-commons-core #3026
e1eb1e2
lorriborri added a commit that referenced this issue May 28, 2024
@lorriborri
Added UploadException and LogSanitazier #3026
75ac56e
lorriborri added a commit that referenced this issue May 28, 2024
@lorriborri
Added UsageException for wrong configurations and download fails #3026
55187d0
lorriborri added a commit that referenced this issue May 29, 2024
@lorriborri
Initial integration test #3026
51e2555
lorriborri added a commit that referenced this issue May 29, 2024
@lorriborri
Added PDS_SORAGE environment variables to be adde to the envrionment …
26da4f6 
…when PDS prepare is called #3026
lorriborri added a commit that referenced this issue May 31, 2024
@lorriborri
Added integrationtest #3026
6054e12
lorriborri added a commit that referenced this issue May 31, 2024
@lorriborri
Added change requests #3026
129e07c
lorriborri added a commit that referenced this issue May 31, 2024
@lorriborri
Removed unnecessary ENV #3026
c0d27f6
de-jcup added a commit that referenced this issue Jun 3, 2024
@de-jcup
Refactoring + changes in validation #3026
ffa77bd 
- validation now without inheritance by introducing input validation
  support class
- renamed some classes - e.g. WrapperSkopeo -> SkopeoWrapper
- changed tests for validation
de-jcup added a commit that referenced this issue Jun 6, 2024
@de-jcup
Refactoring + changes #3026
0a135df 
- validation now without inheritance by introducing input validation
  support class
- renamed some classes - e.g. WrapperSkopeo -> SkopeoWrapper etc.
- changed tests for validation
- removed package prepare.prepare
- moved parts which were used only internally by implementations
  to abstract base class + changed visibility to protected only
- reduced amount of packages, technical parts like factory moved to
  semantic packages or to main prepare package if there are was
  no special semantic
de-jcup added a commit that referenced this issue Jun 7, 2024
@de-jcup
Refactoring + changes #3026
8561617 
- validation now without inheritance by introducing input validation
  support class
- renamed some classes - e.g. WrapperSkopeo -> SkopeoWrapper etc.
- changed tests for validation
- removed package prepare.prepare
- moved parts which were used only internally by implementations
  to abstract base class + changed visibility to protected only
- reduced amount of packages, technical parts like factory moved to
  semantic packages or to main prepare package if there are was
  no special semantic
- refactored tests to make easier to read - e.g. same setup parts
  to beforeEach method etc.
de-jcup added a commit that referenced this issue Jun 7, 2024
@de-jcup
Fixed storage problem and enhanced integration tests #3026
8029f32
de-jcup added a commit that referenced this issue Jun 7, 2024
@de-jcup
Fixed storage problem and enhanced integration tests #3026
2dd0cb7
de-jcup added a commit that referenced this issue Jun 10, 2024
@de-jcup
Refactoring and bugfixes #3026
083b221 
- renamed ToolWrapper to AbstractToolWrapper
- fixed failure in javadoc of Scenario 22
- renamed some parts inside skopeo preparation module
- dropped getProcess() inside ProcessAdapter (avoid that anybody
  uses direct process access - all shall be done by adapter)
- removed pepare wrapper environment variable class, was not used
  any more
- extracted Git location converter and wrote tests
- url conversion method now returns an URL instead of a string
de-jcup added a commit that referenced this issue Jun 10, 2024
@de-jcup
Skopeo call changes #3026
6b535c4 
- replaced process builder call for cleanup on skopeo with
  java file api variant (delete auth file)
- skopeo now called with stdin input for password
- introduced process builder factory + tests
- process adapter supports now user input per stdin
- added test for process adapter support (stdin)
- moved logic for skopeo location conversion to own class
  (SkopeoLocationConverter) and wrote tests, afterwards
  implementation was changed
de-jcup added a commit that referenced this issue Jun 10, 2024
@de-jcup
Changed directory cleanup from process to plain java #3026
5f4a52e 
- introduced DirectoryAndFileSupport, wrote tests
- introduced AutoCleanupGitFilesFilter, wrote tests
- updated GitWrapperTest
- separated former git auto clean environment entry to
  KEY_PDS_PREPARE_MODULE_GIT_REMOVE_GIT_FILES_BEFORE_UPLOAD
  KEY_PDS_PREPARE_MODULE_GIT_CLONE_WITHOUT_GIT_HISTORY
- changed environment entries for skopeo as well, everything from
  a module is now prefixed "pds.prepare.module." etc.
de-jcup added a commit that referenced this issue Jun 10, 2024
@de-jcup
Changed directory cleanup from process to plain java #3026
d8bcdf8 
- introduced DirectoryAndFileSupport, wrote tests
- introduced AutoCleanupGitFilesFilter, wrote tests
- updated GitWrapperTest
- separated former git auto clean environment entry to
  KEY_PDS_PREPARE_MODULE_GIT_REMOVE_GIT_FILES_BEFORE_UPLOAD
  KEY_PDS_PREPARE_MODULE_GIT_CLONE_WITHOUT_GIT_HISTORY
- changed environment entries for skopeo as well, everything from
  a module is now prefixed "pds.prepare.module." etc.
de-jcup added a commit that referenced this issue Jun 11, 2024
@de-jcup
Changed directory cleanup from process to plain java #3026
63ce627 
- introduced DirectoryAndFileSupport, wrote tests
- introduced AutoCleanupGitFilesFilter, wrote tests
- updated GitWrapperTest
- separated former git auto clean environment entry to
  KEY_PDS_PREPARE_MODULE_GIT_REMOVE_GIT_FILES_BEFORE_UPLOAD
  KEY_PDS_PREPARE_MODULE_GIT_CLONE_WITHOUT_GIT_HISTORY
- changed environment entries for skopeo as well, everything from
  a module is now prefixed "pds.prepare.module." etc.
de-jcup added a commit that referenced this issue Jun 11, 2024
@de-jcup
Changed git prepare module cleanup behavior and documentatino #3026
30e7e64 
- when git history is not wanted, the .git folder is now
  always removed
- additional gitfiles are treated extra
- pds config file in pds prepare solution has now the parameters
  inside as optional
de-jcup added a commit that referenced this issue Jun 13, 2024
@de-jcup
Review changes #3026
87d2b9c 
- added PDS_WORKSPACE_AUTOCLEAN_DISABLED handling in deployment.yaml
  again
de-jcup added a commit that referenced this issue Jun 13, 2024
@de-jcup
Review changes #3026
a1b1809 
- dropped unnecessary class
- added missing licence header
- small refactoring inside DirectoryAndFileSupport
de-jcup added a commit that referenced this issue Jun 13, 2024
@lorriborri @de-jcup
PDS prepare upload remote data to shared storage (#3131)
1617be0 
* Integrated upload classes and archive support for prepare #3026
* Moved LogSanitizer in pds-commons-core #3026
* Fixed storage problem and enhanced integration tests #3026
- fixed failure in javadoc of Scenario 22
* Skopeo call changes #3026
- replaced process builder call for cleanup on skopeo with
  java file api variant (delete auth file)
- skopeo now called with stdin input for password
- introduced process builder factory + tests
- process adapter supports now user input per stdin
- added test for process adapter support (stdin)
- moved logic for skopeo location conversion to own class
  (SkopeoLocationConverter) and wrote tests, afterwards
  implementation was changed

* Changed directory cleanup from process to plain java #3026
- introduced DirectoryAndFileSupport, wrote tests
- introduced AutoCleanupGitFilesFilter, wrote tests
- updated GitWrapperTest
- separated former git auto clean environment entry to
  KEY_PDS_PREPARE_MODULE_GIT_REMOVE_GIT_FILES_BEFORE_UPLOAD
  KEY_PDS_PREPARE_MODULE_GIT_CLONE_WITHOUT_GIT_HISTORY
- changed environment entries for skopeo as well, everything from
  a module is now prefixed "pds.prepare.module." etc.

* Changed git prepare module cleanup behavior and documentatino #3026
- when git history is not wanted, the .git folder is now
  always removed
- additional gitfiles are treated extra
- pds config file in pds prepare solution has now the parameters
  inside as optional

---------

Co-authored-by: Albert Tregnaghi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lorriborri lorriborri

Labels
prepare
Projects
None yet
Milestone
Prepare Wrapper 1.0.0
Development

Successfully merging a pull request may close this issue.

PDS prepare upload remote data to shared storage mercedes-benz/sechub
2 participants
@de-jcup @lorriborri