Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Rewrite data cp implementation #506

Merged
merged 48 commits into from
May 27, 2022
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
7f924ea
refactor: Rewrite data cp implementation
Apr 29, 2022
d67ab99
feat: Add WebCopier and LocalFSCopier
Apr 29, 2022
0c3ec69
feat: Use LocalToLocalCopier when needed
Apr 29, 2022
dcb0d43
feat: Add check for both source and destination being archives
Apr 29, 2022
5313d6a
feat: Create scaffolding for extract -> compress implementation
Apr 29, 2022
5ef419f
feat: Add suport for recompression in local-to-local copy
May 3, 2022
696a1d5
feat: Add suport for recompression in local-to-cloud copy
May 3, 2022
27627db
feat: Add suport for recompression in cloud-to-local copy
May 3, 2022
ee1eb9b
feat: Add suport platform-to-platform copy (between storage abd disks…
May 3, 2022
6b6233d
feat: Enable command execution
May 3, 2022
084912b
fix: Remove cached_property decorator due to missing python3.7 support
May 3, 2022
3342dab
tests: Enable quickfail in pytest
May 3, 2022
ef4a507
fix: Proper order for _ensure_can_execute() check
May 3, 2022
7fe4d3c
feat: Skip compression if source is already an archive of the same type
May 3, 2022
c216eb0
test: Disable quickfail
May 3, 2022
8f55e7a
fix: Fix volume passing
May 3, 2022
d398bbe
fix: Fix argument passing to .jobs.start()
May 3, 2022
d4f570c
fix: Fix type for job preset
May 3, 2022
ba68649
fix: Add proper mode for storage and disks
May 3, 2022
8014071
fix: Add support for relative paths in disks
May 3, 2022
d10731a
fix: Add support for relative paths in disks
May 4, 2022
0d39523
fix: Pass config to job; ensure tmp directory exists
May 4, 2022
93e5abc
fix: Pin click==8.1.2
May 4, 2022
6bf6671
fix: Fix file mounts on destination for storage resources
May 4, 2022
ef7f3b8
fix: Ensure destination directory exists when extracting
May 4, 2022
6c1aee2
test: Pin click in tests
May 4, 2022
7ddf9d8
refactor: Simplify code for data copy; improve docs
May 5, 2022
5bbf438
docs: Add changelog; fix help for -t flag
May 5, 2022
f79db81
ci: Increase test duration to 45min
andriihomiak May 5, 2022
55d2525
refactor: Apply PR review suggestions
andriihomiak May 12, 2022
618c3a9
merge: Merge branch 'refactor/data_cp' of github.com:neuro-inc/neuro-…
andriihomiak May 12, 2022
6d3265b
test: Stop testing use_temp_dir
andriihomiak May 12, 2022
99a6fac
test: `data cp` to cloud (#515)
andriihomiak May 23, 2022
73b8562
test: Remove unnecesary retry
andriihomiak May 23, 2022
fd16a90
test: Change number of workers
andriihomiak May 23, 2022
e974646
test: Skip cloud-to-local and local-to-cloud tests; increase timeout …
andriihomiak May 23, 2022
d90d265
misc: Log only id of the job
andriihomiak May 24, 2022
3d62bb0
test: Increase test timeout to 90min
andriihomiak May 24, 2022
f3a913f
feat: Use smaller default preset for data copy
andriihomiak May 24, 2022
2ce07ce
test: Decrease paralellism
andriihomiak May 24, 2022
f23ff12
ci: Limit jobs
andriihomiak May 24, 2022
710e656
test: Increase retry count and enable fail-fast
andriihomiak May 24, 2022
4668b25
test: Increase retry count and enable fail-fast
andriihomiak May 24, 2022
5ad162e
test: Decrease retry and remove time limit on retries
andriihomiak May 24, 2022
988732c
test: Add wait between failed tests
andriihomiak May 24, 2022
91897b5
test: Mark flaky tests with xfail
andriihomiak May 25, 2022
9238515
ci: Remove fail-fast and parallel job limit
andriihomiak May 25, 2022
992638d
ci: Run only 1 ci job at once
andriihomiak May 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,16 @@ ARG NEURO_EXTRAS_PACKAGE=neuro-extras

ENV PATH=/root/.local/bin:$PATH

RUN pip3 install --no-cache-dir -U pip pipx
RUN pip3 install --no-cache-dir -U pip pipx click==8.1.2 # TODO remove click pinned version
RUN MULTIDICT_NO_EXTENSIONS=1 YARL_NO_EXTENSIONS=1 pip install --user \
$NEURO_EXTRAS_PACKAGE && \
# neuro-flow is used in outforz, not in reqs file since NF itself requires NE
pipx install neuro-flow && \
pipx install awscli # isolated env since it has conflicts with neuro-cli
# isolated env since it has conflicts with neuro-cli
pipx install awscli && \
# pipx reinstallation is required untill https://github.com/neuro-inc/neuro-cli/pull/2671 is resolved
pipx runpip neuro-flow uninstall -y click && \
pipx runpip neuro-flow install "click==8.1.2"
RUN neuro-extras init-aliases

RUN mkdir -p /root/.ssh
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
COLOR ?= auto
PYTEST_FLAGS = -v
PYTEST_FLAGS = -v -x
PYTEST_PARALLEL = 5

.PHONY: setup
Expand Down
3 changes: 2 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@ neuro-extras data cp [OPTIONS] SOURCE DESTINATION
| _-c, --compress_ | Perform compression of SOURCE into the DESTINATION file. The archive type is derived from the file name. Supported types: .tar.gz, .tgz, .tar.bz2, .tbz2, .tbz, .tar, .gz, .zip. |
| _-v, --volume MOUNT_ | Mounts directory from vault into container. Use multiple options to mount more than one volume. |
| _-e, --env VAR=VAL_ | Set environment variable in container. Use multiple options to define more than one variable. |
| _-t, --use-temp-dir_ | Download and extract / compress data \(if needed\) inside the temporal directory. Afterwards move resulted file\(s\) into the DESTINATION. NOTE: use it if 'storage:' is involved and extraction or compression is performed to speedup the process. |
| _-t, --use-temp-dir_ | DEPRECATED - need for temp dir is automatically detected, this flag will be removed in a future release.
Download and extract / compress data \(if needed\) inside the temporal directory. Afterwards move resulted file\(s\) into the DESTINATION. NOTE: use it if 'storage:' is involved and extraction or compression is performed to speedup the process. |
| _-s, --preset PRESET\_NAME_ | Preset name used for copy. |
| _-l, --life\_span SECONDS_ | Copy job life span in seconds. |
| _--help_ | Show this message and exit. |
Expand Down
Loading