[WIP] Implement GroupQueryAttention from ORT#654

Draft

jslhcl wants to merge 19 commits intomainfrom leca/groupQueryAttention

+13,966-7

Commits on Feb 9, 2024

Commits on Feb 22, 2024

Commits on Feb 27, 2024

add test case and override GetInputMemoryType
jslhcl
committed

Commits on Mar 5, 2024

UT work, but need to hardcode CMAKE_CUDA_ARCHITECTURES in cmake
jslhcl
committed

Commits on Mar 11, 2024

add result compare. Flash attention needs to add -DCMAKE_CUDA_ARCHITECTURES=86 in cmdclass.py
Lei Cao
committed

Commits on Mar 14, 2024

optimization works
jslhcl
committed

Commits on Apr 8, 2024

Commits on Apr 25, 2024

merge from main
Cheng Tang
committed

Commits on Apr 28, 2024

make the v2 struct kernel been able to be called from eager mode
Cheng Tang
committed
remove legacy kernel context
Cheng Tang
committed
update FastGelu
Cheng Tang
committed

Commits on Apr 29, 2024

add fast gelu test
Cheng Tang
committed

Commits on Apr 30, 2024

Merge branch 'main' into leca/groupQueryAttention
Cheng Tang
committed

Commits on May 1, 2024

move attention ops to cuda/attention-li
Cheng Tang
committed

Commits on May 2, 2024

a temp fix
Cheng Tang
committed
add GQA test
Cheng Tang
committed