Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

whole_update_double_buffer.cpp failing in CI on CUDA #13731

Closed
sarnex opened this issue May 9, 2024 · 2 comments
Closed

whole_update_double_buffer.cpp failing in CI on CUDA #13731

sarnex opened this issue May 9, 2024 · 2 comments
Labels
bug Something isn't working cuda CUDA back-end sycl-graph

Comments

@sarnex
Copy link
Contributor

sarnex commented May 9, 2024

Describe the bug

FAIL: SYCL :: Graph/Update/Explicit/whole_update_double_buffer.cpp (1108 of 2032)
******************** TEST 'SYCL :: Graph/Update/Explicit/whole_update_double_buffer.cpp' FAILED ********************
Exit Code: -6

Command Output (stdout):
--
# RUN: at line 1
/__w/llvm/llvm/toolchain/bin//clang++   -fsycl -fsycl-targets=nvptx64-nvidia-cuda  /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/Update/Explicit/whole_update_double_buffer.cpp -o /__w/llvm/llvm/build-e2e/Graph/Update/Explicit/Output/whole_update_double_buffer.cpp.tmp.out
# executed command: /__w/llvm/llvm/toolchain/bin//clang++ -fsycl -fsycl-targets=nvptx64-nvidia-cuda /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/Update/Explicit/whole_update_double_buffer.cpp -o /__w/llvm/llvm/build-e2e/Graph/Update/Explicit/Output/whole_update_double_buffer.cpp.tmp.out
# note: command had no output on stdout or stderr
# RUN: at line 2
env SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1 ONEAPI_DEVICE_SELECTOR=cuda:gpu  /__w/llvm/llvm/build-e2e/Graph/Update/Explicit/Output/whole_update_double_buffer.cpp.tmp.out
# executed command: env SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1 ONEAPI_DEVICE_SELECTOR=cuda:gpu /__w/llvm/llvm/build-e2e/Graph/Update/Explicit/Output/whole_update_double_buffer.cpp.tmp.out
# .---command stdout------------
# | Unexpected value at index 256 for DataB2: 1305 (got) vs 1574 (expected)
# `-----------------------------
# .---command stderr------------
# | whole_update_double_buffer.cpp.tmp.out: /__w/llvm/llvm/llvm/sycl/test-e2e/Graph/Update/Explicit/../../Inputs/whole_update_double_buffer.cpp:86: int main(): Assertion `check_value(i, ReferenceB2[i], HostDataB2[i], "DataB2")' failed.
# `-----------------------------
# error: command failed with exit status: -6

--

********************
Slowest Tests:

from this PR with run here.

To reproduce

No response

Environment

No response

Additional context

No response
@sarnex sarnex added bug Something isn't working cuda CUDA back-end labels May 9, 2024
@sarnex sarnex mentioned this issue May 9, 2024
Merged
@EwanC
Copy link
Contributor

EwanC commented May 10, 2024

This came up in #13719 (comment) too, and has also been reported internally as failing on Windows runs. Will disable the test on CUDA while @intel/sycl-graphs-reviewers investigate.

EwanC added a commit to reble/llvm that referenced this issue May 10, 2024
@EwanC
[SYCL][Graph][CUDA] Disable failing update test.
80e01b4 
intel#13731

Disable whole graph update tests on CUDA, as they are
flaky and failing on unrelated PRs, until such a time as
the fail can be investigated and fixed.
@EwanC EwanC mentioned this issue May 10, 2024
Merged
ldrumm pushed a commit that referenced this issue May 10, 2024
@EwanC
[SYCL][Graph][CUDA] Disable failing update test. (#13741)
0dfd5e4 
#13731

Disable whole graph update tests on CUDA, as they are flaky and failing
on unrelated PRs, until such a time as the fail can be investigated and
fixed.
@Bensuo Bensuo mentioned this issue Jun 13, 2024
Merged
sommerlukas pushed a commit that referenced this issue Jun 19, 2024
@Bensuo
[SYCL][Graph] Wait instead of flush dep events in update command (#14167
4151c79 
)

- Could cause overlap of graph execution and initial h2d copy when
updated with previously unused buffers
- Re-enable update double buffer tests on CUDA as this should fix
failures observed in #13731
@Bensuo
Copy link
Contributor

Bensuo commented Jun 19, 2024

This should be fixed by #14167

@EwanC EwanC closed this as completed Jun 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working cuda CUDA back-end sycl-graph
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@EwanC @npmiller @sarnex @Bensuo