Enhanced gemma prediction with new flawless logit #51
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PiperOrigin-RevId: 663277444 Change-Id: I8d7030ce586577a433c48f32df7efa7c141b171a
…ormer_lib.make_causal_attn_mask(input_mask)` PiperOrigin-RevId: 663692225 Change-Id: Ie2cb6229302087ea1ce5b5c7f442a088207ead07
PiperOrigin-RevId: 665414923 Change-Id: I42bc41074518e3065f85c7f1a3014fdd09cffe4c
Currently all weights in FeedForward layers are initialized to zero. This doesn't cause any issues when loading the module with pretrained weights, but if training from scratch it will result in all gradients being zero throughout training so no learning can occur. Changing w_gating be be initialized from a normal distribution fixes this. PiperOrigin-RevId: 674306730 Change-Id: I90800dbe605cdf88f341d103f102357ff278a393
PiperOrigin-RevId: 674394389 Change-Id: I25ba5ad4769c3101c2bf572e33723d4a241e3895
…se errors for implicit rank promotion. PiperOrigin-RevId: 675179053 Change-Id: I55459c1aa99c7d33ae3f03712eaed01ccc5fc9f2
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
The GitHub CLA check doesn't recognize the noreply user @a-googler <no****ly@google.com>. How shall I proceed? Should I use an interactive rebase to edit the author of the related commits? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integration of a
flawless_logitBy normalizing the logits for each token, the model ensures that the predictions are more balanced and less likely to be dominated by any single token.
Subtracting the normalized sum can help reduce biases and make the logits more representative of the actual distribution of the data.
Based on initial tests conducted on Gemma2 7B, it appears that the performance at inference time has been improved.