Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate ttnn resnet 50 batch_size 8 failure on fast dispatch nightly #8555

Open
tt-rkim opened this issue May 16, 2024 · 3 comments
Open

Investigate ttnn resnet 50 batch_size 8 failure on fast dispatch nightly #8555

tt-rkim opened this issue May 16, 2024 · 3 comments
Assignees
@mywoodstock @nsmithtt
Labels
ci-bug bugs found in CI P1_critical ResNet-50

Comments

@tt-rkim
Copy link
Collaborator

tt-rkim commented May 16, 2024

This has been failing for a couple of days:

FAILED tests/ttnn/integration_tests/resnet/test_ttnn_functional_resnet50.py::test_resnet_50[batch_size=8-act_dtype=DataType.BFLOAT8_B-weight_dtype=DataType.BFLOAT8_B-math_fidelity=MathFidelity.LoFi-device_l1_small_size=24576] - IndexError: _Map_base::at

Skipping for now

@mywoodstock and @nsmithtt are in the know
@tt-rkim tt-rkim added P1_critical ci-bug bugs found in CI labels May 16, 2024
tt-rkim added a commit that referenced this issue May 16, 2024
@tt-rkim
#8555: Skip batch 8 resnet50 in ttnn because it's failing and being l…
174aa78 
…ooked at
tt-rkim added a commit that referenced this issue May 16, 2024
@tt-rkim
#8555: Skip batch 8 resnet50 in ttnn because it's failing and being l…
decf342 
…ooked at
tt-rkim added a commit that referenced this issue May 16, 2024
@tt-rkim
#8555: Skip batch 8 resnet50 in ttnn because it's failing and being l…
de9a19b 
…ooked at (#8556)
@tarafdarTT tarafdarTT mentioned this issue May 16, 2024
Closed
@tt-rkim
Copy link
Collaborator Author

tt-rkim commented May 17, 2024

https://github.com/tenstorrent/tt-metal/actions/runs/9131122193
Re-running on branch on @nsmithtt 's re-enable commit, because I'm seeing this locally on my GS machine

@nsmithtt
Copy link
Contributor

The error is coming from resharding from height to block, related to #8260, with this fix along with another fix similar to Vraj's #8462 the issue goes away.

@tt-rkim
Copy link
Collaborator Author

tt-rkim commented May 23, 2024

Sounds great, let's close this issue once we get those two diffs in and a green pass

@nsmithtt nsmithtt mentioned this issue May 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mywoodstock mywoodstock

@nsmithtt nsmithtt

Labels
ci-bug bugs found in CI P1_critical ResNet-50
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mywoodstock @tt-rkim @nsmithtt