-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Questions about KAdaptation implementation #6
Comments
Hi, many thanks for interests! The scaling factor is a hyper-parameter, you can manually adjust it but from my experience it won't affect the performance much. For the value matrix, actually we share the same decomposition here so that's why reusing it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi, thanks for the great work and releasing the code to reproduce it.
I have a few questions regarding the kronecker adaptation forward pass through the adapter modules:
(1) The scaling factor you use for the KAdaptation is 1/5 times the scaling used in standard LoRA:
PEViT/vision_benchmark/evaluation/model.py
Line 564 in be6fb43
Is there a justification for this or is it simply an empirical magic number?
(2) While forwarding through your adapter for the value matrix, it seems like you reuse the query weight matrix (A as defined in the paper as I understand it). Is this a typo/bug?
PEViT/vision_benchmark/evaluation/model.py
Lines 571 to 580 in be6fb43
Shouldn't line 580 be
H = kronecker_product_einsum_batched(phm_rule2, Wv).sum(0)
instead?The text was updated successfully, but these errors were encountered: