feature(wrh): add RoPE for unizero #263
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
puyuan1996
reviewed
Aug 13, 2024
| @@ -55,6 +55,15 @@ def __init__(self, config: TransformerConfig) -> None: | |||
| self.blocks = nn.ModuleList([Block(config) for _ in range(config.num_layers)]) | |||
| self.ln_f = nn.LayerNorm(config.embed_dim) | |||
|
|
|||
| self.config.rope_theta = 500000 | |||
| self.config.max_seq_len = 2048 | |||
There was a problem hiding this comment.
这个参数确定一下，是否应该设置成与实际训练的长度一致
There was a problem hiding this comment.
rope_theta 是影响位置编码的频率，应该就用默认的就行。max_seq_len 是最大序列长度，它决定了预计算频率张量的长度，如果我们希望在测试时支持更长的序列，应该将 max_seq_len 设置为能覆盖我们期望的最大测试序列长度，例如如果我们测试最长是2048，这个值应该设置为2048， 10有点太小了，如果测试长度>10会报错。
puyuan1996
reviewed
Aug 13, 2024
| max_env_step = int(5e5) | ||
| reanalyze_ratio = 0. | ||
| batch_size = 2 | ||
| num_unroll_steps = 10 |
There was a problem hiding this comment.
训练的时候，不是用的debug config吧
There was a problem hiding this comment.
不是，只是提交上来的是debug版的， 但Training的过程里用的不是
|
This PR will be updated in #266. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We add RoPE for unizero