h2o for kv cache compression #1468

n1ck-guo · 2024-04-10T07:37:20Z

Type of Change

feature

Description

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
paper

NTD

Expected Behavior & Potential Risk

None

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

Signed-off-by: n1ck-guo <[email protected]>

github-actions · 2024-04-10T07:37:44Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 Format Scan Tests workflow

Check ID	Status
format-scan (pylint)	success	✅
format-scan (bandit)	success	✅
format-scan (cloc)	success	✅
format-scan (cpplint)	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

🟢 Optimize Unit Test workflow

Check ID	Status
optimize-unit-test-baseline	success	✅
optimize-unit-test-PR-test	success	✅
Genreate-OptimizeUT-Report	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

🟢 Engine Unit Test workflow

Check ID	Status
engine-unit-test-baseline	success	✅
engine-unit-test-PR-test	success	✅
Genreate-Engine-Report	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact VincyZhang or XuehaoSun for help.

for more information, see https://pre-commit.ci

...ion_for_transformers/transformers/modeling/kv_cahe_compression/h2o_real_drop/modify_llama.py

...on_for_transformers/transformers/modeling/kv_cahe_compression/h2o_sim_drop/modify_gptneox.py

Signed-off-by: biao.fang <[email protected]>

for more information, see https://pre-commit.ci

intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

...nsion_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py

examples/huggingface/pytorch/text-generation/h2o/run_lm_eval_harness.py

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: biao.fang <[email protected]>

Signed-off-by: n1ck-guo <[email protected]>

PenghuiCheng · 2024-07-02T05:27:11Z

Could we add a document introducing what h2o is?

changwangss · 2024-07-02T09:18:00Z

format scan improved by #1647. merged.

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

n1ck-guo · 2024-07-15T02:51:33Z

Could we add a document introducing what h2o is?

add in the example/readme

Signed-off-by: n1ck-guo <[email protected]>

…n-for-transformers into hengguo/h2o

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

h2o for kv cache compression

41d8647

Signed-off-by: n1ck-guo <[email protected]>

n1ck-guo requested review from PenghuiCheng, VincyZhang and wenhuach21 April 10, 2024 07:37

[pre-commit.ci] auto fixes from pre-commit.com hooks

eb7f564

for more information, see https://pre-commit.ci

wenhuach21 reviewed Apr 10, 2024

View reviewed changes

...ion_for_transformers/transformers/modeling/kv_cahe_compression/h2o_real_drop/modify_llama.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Apr 10, 2024

View reviewed changes

...ion_for_transformers/transformers/modeling/kv_cahe_compression/h2o_real_drop/modify_llama.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Apr 10, 2024

View reviewed changes

...on_for_transformers/transformers/modeling/kv_cahe_compression/h2o_sim_drop/modify_gptneox.py Outdated Show resolved Hide resolved

BiaoFangAIA and others added 3 commits April 23, 2024 16:36

rebuild

c46ea7d

Signed-off-by: biao.fang <[email protected]>

merge

95ff9ae

Signed-off-by: biao.fang <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9d27733

for more information, see https://pre-commit.ci

wenhuach21 reviewed Apr 25, 2024

View reviewed changes

intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py Outdated Show resolved Hide resolved

n1ck-guo and others added 4 commits April 25, 2024 03:01

update

444490d

Signed-off-by: n1ck-guo <[email protected]>

update

4309089

Signed-off-by: n1ck-guo <[email protected]>

merge

8c5272e

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

1b83e52

for more information, see https://pre-commit.ci

wenhuach21 reviewed Apr 29, 2024

View reviewed changes

...nsion_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Apr 29, 2024

View reviewed changes

examples/huggingface/pytorch/text-generation/h2o/run_lm_eval_harness.py Outdated Show resolved Hide resolved

n1ck-guo and others added 3 commits May 7, 2024 04:26

update

3fd73cb

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

a2d3ae0

for more information, see https://pre-commit.ci

Merge branch 'main' into hengguo/h2o

ddf5445

VincyZhang added the WIP label May 13, 2024

n1ck-guo and others added 7 commits May 14, 2024 01:43

Merge branch 'main' into hengguo/h2o

91d4394

real drop

a83e6d6

Signed-off-by: n1ck-guo <[email protected]>

modify real drop code

92c8a62

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

70a1cf3

for more information, see https://pre-commit.ci

fix

bc9eade

Signed-off-by: biao.fang <[email protected]>

update for real drop and sim mode, using the same api

9aa25f6

Signed-off-by: n1ck-guo <[email protected]>

Merge branch 'main' into hengguo/h2o

03cdc8d

changwangss and others added 2 commits July 3, 2024 17:08

Merge branch 'main' into hengguo/h2o

cedcd43

Merge branch 'main' into hengguo/h2o

eb8441c

changwangss approved these changes Jul 12, 2024

View reviewed changes

n1ck-guo and others added 3 commits July 15, 2024 10:16

Merge branch 'main' into hengguo/h2o

0894b6d

add desc to h2o in readme

d241c25

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

0c547c5

for more information, see https://pre-commit.ci

n1ck-guo and others added 3 commits July 15, 2024 01:36

add doc for h2o

3723158

Signed-off-by: n1ck-guo <[email protected]>

Merge branch 'hengguo/h2o' of https://github.com/intel/intel-extensio…

7da0cf5

…n-for-transformers into hengguo/h2o

[pre-commit.ci] auto fixes from pre-commit.com hooks

d600112

for more information, see https://pre-commit.ci

n1ck-guo requested a review from XuehaoSun July 15, 2024 05:44

n1ck-guo added the ready for merge label Jul 15, 2024

PenghuiCheng approved these changes Jul 15, 2024

View reviewed changes

n1ck-guo removed the request for review from VincyZhang July 16, 2024 01:29

update

b1ab771

Signed-off-by: n1ck-guo <[email protected]>

n1ck-guo requested a review from VincyZhang as a code owner July 16, 2024 04:49

n1ck-guo and others added 9 commits July 16, 2024 02:10

add ut

4bee0f0

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3fad641

for more information, see https://pre-commit.ci

Merge branch 'main' into hengguo/h2o

46fa4d7

fix

fda316d

Signed-off-by: n1ck-guo <[email protected]>

Merge branch 'main' into hengguo/h2o

268fb80

update

b7a1bd1

Signed-off-by: n1ck-guo <[email protected]>

merge

2ae387a

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

2706d5d

for more information, see https://pre-commit.ci

Merge branch 'main' into hengguo/h2o

b2d5768

n1ck-guo removed the request for review from yintong-lu July 29, 2024 05:52

XuehaoSun merged commit 3e78ae8 into main Jul 29, 2024
17 checks passed

XuehaoSun deleted the hengguo/h2o branch July 29, 2024 05:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

h2o for kv cache compression #1468

h2o for kv cache compression #1468

n1ck-guo commented Apr 10, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

PenghuiCheng commented Jul 2, 2024

changwangss commented Jul 2, 2024 •

edited

Loading

n1ck-guo commented Jul 15, 2024

h2o for kv cache compression #1468

h2o for kv cache compression #1468

Conversation

n1ck-guo commented Apr 10, 2024 • edited Loading

Type of Change

Description

Expected Behavior & Potential Risk

How has this PR been tested?

Dependency Change?

github-actions bot commented Apr 10, 2024 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

PenghuiCheng commented Jul 2, 2024

changwangss commented Jul 2, 2024 • edited Loading

n1ck-guo commented Jul 15, 2024

n1ck-guo commented Apr 10, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

changwangss commented Jul 2, 2024 •

edited

Loading