Layers that are never used in ByteLatentTransformer class

[isayoften]

Hello! Please check this line :

blt/bytelatent/model/blt.py

Line 834 in 6ffeb66

self.layers = nn.ModuleList(

It seems that these layers are here by mistake. All transformerblocks are defined inside the local and global parts. There is no self.layers in the forward at all also

[isayoften]

The same question about:

blt/bytelatent/model/blt.py

Line 831 in 6ffeb66

self.tok_embeddings = torch.nn.Embedding(args.vocab_size, args.dim)

and:

blt/bytelatent/model/blt.py

Line 854 in 6ffeb66

self.output = nn.Linear(args.dim, args.vocab_size, bias=False)

because we already have tok_embeds and unembeds in encoder and decoder:

blt/bytelatent/model/local_models.py

Line 81 in 6ffeb66

self.tok_embeddings = nn.Embedding(self.vocab_size, args.dim)

blt/bytelatent/model/local_models.py

Line 330 in 6ffeb66

self.output = nn.Linear(

[isayoften]

Also, we initialise tok_embeds twice because of super() call in the decoder:

blt/bytelatent/model/local_models.py

Line 308 in 6ffeb66

super().__init__(args)