Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Organize full model integration tests in some central location #49

Open
ScottTodd opened this issue Nov 26, 2024 · 0 comments
Open

Organize full model integration tests in some central location #49

ScottTodd opened this issue Nov 26, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@ScottTodd
Copy link
Member

These two locations have full model integration tests:

Those tests are both currently run as part of the https://github.com/iree-org/iree/blob/main/.github/workflows/pkgci_regression_test.yml workflow.

Together with #48, it should be clear where new tests should go in source code and within workflows in this repository and in iree-org/iree.

The documentation at https://iree.dev/developers/general/testing-guide/#external-test-suites should also be kept up to date.

File storage

Both of those test suites currently store files in the 'sharkpublic' Azure storage account.

  • For some models we should be able to download from https://huggingface.co/models instead, though the import process into .mlir may involve using packages outside of iree-org like https://github.com/nod-ai/shark-ai.
  • If the community is comfortable with files that need special permissions to maintain, we could continue using that storage account
  • We could see if the Linux Foundation has storage we could use somehow, and grant access to that using iree-org GitHub membership or some other open process

Test flags

The tests at https://github.com/iree-org/iree/tree/main/experimental/regression_suite/shark-test-suite-models in particular have long lists of flags and hardware requirements since most model development activities have been focusing on getting great performance on a small number of devices (e.g. AMD MI300 GPUs). For a common test suite, I would prefer to use default or generic flags, not long lists of flags, transform scripts, or tuning specs.

@ScottTodd ScottTodd added the enhancement New feature or request label Dec 11, 2024
ScottTodd added a commit to nod-ai/SHARK-TestSuite that referenced this issue Dec 19, 2024
These tests are redundant with newer test suites here:

*
https://github.com/iree-org/iree/tree/main/experimental/regression_suite/shark-test-suite-models
(sd3 and sdxl)
* Note that these tests use files from
https://huggingface.co/amd-shark/sdxl-quant-models, while the tests
there use Azure (sharkpublic)
*
https://github.com/iree-org/iree-test-suites/tree/main/sharktank_models
(llama)
* Note that llama 3 tested there is different from the
[openlm-research/open_llama_3b_v2](https://huggingface.co/openlm-research/open_llama_3b_v2)
model tested here, but we should move tests to iree-test-suites anyways:
iree-org/iree-test-suites#49
ScottTodd added a commit to nod-ai/SHARK-TestSuite that referenced this issue Jan 8, 2025
The iree_tests folder has been migrated to a new repository at
https://github.com/iree-org/iree-test-suites, see
iree-org/iree-test-suites#49. The only tests
remaining here have been marked XFAIL since
iree-org/iree#17344 because the files needed
to be regenerated:
```json
    "expected_compile_failures": [
      "pytorch/models/opt-125M",
      "pytorch/models/resnet50",
    ],
```

See also iree-org/iree#19614, which stops
cloning this repository and running these tests in
https://github.com/iree-org/iree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant