Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve support for remote executors #1885

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

cjao
Copy link
Contributor

@cjao cjao commented Dec 12, 2023

  • I have added the tests to cover my changes.
  • I have updated the documentation and CHANGELOG accordingly.
  • I have read the CONTRIBUTING document.

The changes in this PR allow remote executors to use the new-style executor API.

  • The file transfer strategy can now be inferred automatically for various URI scheme pairings. For example, (s3, file) and (file, s3) both map to the S3 transfer strategy.
  • Introduced framework for building sublattices inside executors. If the COVALENT_STAGING_URI_PREFIX environment variable is set (for instance, COVALENT_STAGING_URI_PREFIX="s3://{bucket_name}/{dispatch_id}/sublattices", the executor will deposit the sublattice assets there for the orchestrator to retrieve after the _build_sublattice_graph task completes (naturally Covalent should have read permissions for that URI prefix).

Copy link

codecov bot commented Dec 12, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.24%. Comparing base (d7841c7) to head (3a0aa15).
Report is 1 commits behind head on develop.

Current head 3a0aa15 differs from pull request most recent head 4aca466

Please upload reports for the commit 4aca466 to get more accurate results.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1885      +/-   ##
===========================================
+ Coverage    84.60%   85.24%   +0.63%     
===========================================
  Files          295      181     -114     
  Lines        14499    11328    -3171     
  Branches       195        0     -195     
===========================================
- Hits         12267     9656    -2611     
+ Misses        2098     1672     -426     
+ Partials       134        0     -134     
Flag Coverage Δ
Dispatcher 92.55% <100.00%> (-0.04%) ⬇️
Functional_Tests ?
SDK 79.75% <100.00%> (+0.14%) ⬆️
UI_Backend ?
UI_Frontend ?

@cjao cjao marked this pull request as ready for review December 12, 2023 13:37
@cjao cjao requested a review from a team as a code owner December 12, 2023 13:37
@cjao cjao marked this pull request as draft December 12, 2023 13:38
@cjao cjao force-pushed the improve-file-transfer-selector branch from 72beaea to 7fac492 Compare December 13, 2023 21:52
@cjao cjao marked this pull request as ready for review December 15, 2023 16:26
@cjao cjao requested a review from a team as a code owner December 15, 2023 16:26
@cjao cjao force-pushed the improve-file-transfer-selector branch 3 times, most recently from 678996e to c254d46 Compare January 23, 2024 12:27
@cjao cjao force-pushed the improve-file-transfer-selector branch 3 times, most recently from 04fde96 to 1313c44 Compare February 5, 2024 13:18
@cjao cjao force-pushed the improve-file-transfer-selector branch 2 times, most recently from 45bb7e2 to c083aa9 Compare February 16, 2024 01:55
@cjao cjao force-pushed the improve-file-transfer-selector branch from c083aa9 to 3a0aa15 Compare February 29, 2024 13:24
@cjao cjao force-pushed the improve-file-transfer-selector branch from 3a0aa15 to 7ec3c58 Compare May 16, 2024 00:11
@cjao cjao force-pushed the improve-file-transfer-selector branch 12 times, most recently from 139d21a to 4aca466 Compare May 27, 2024 18:48
Handle the following cases:
- Local-Remote (except HTTP destination)
- Remote-Local
- Local-Local

This will be needed for to port remote executors to the
new runner-executor API.
`LocalDispatcher.prepare_manifest()` will attempt to copy the task
artifacts to `COVALENT_STAGING_URI_PREFIX` if that environment
variable is set.
@cjao cjao force-pushed the improve-file-transfer-selector branch from 4aca466 to 40cfdb8 Compare September 11, 2024 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant