ACNN implementation has only 1 Conv layer

[gerardsimons]

I think something is wrong with the ACNN implementation as the entire CNN exists out of a single Conv1d layer:

# (batch, channels, length)
self.cnn = nn.Conv1d(in_channels=self.in_channels, 
                    out_channels=self.out_channels, 
                    kernel_size=16, 
                    stride=4)

[hsd1503]

The attention mechanism (self attention) is written in Line 102 with raw torch.matmul implementations.

[gerardsimons]

Yes, but shouldn't there still be more than 1 ConvLayer? I would expect a stack of Conv blocks of at least 10+ after which the transformer layers would happen, no?

If I run test_physionet_acnn.py as it is I get the following summary. I understand that the attention may not be printed in the summary bu there must be more CNN blocks?

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv1d-1               [-1, 128, 9]           2,176
            Linear-2                    [-1, 4]             516
================================================================
Total params: 2,692
Trainable params: 2,692
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.01
Forward/backward pass size (MB): 0.01
Params size (MB): 0.01
Estimated Total Size (MB): 0.03

[hsd1503]

Just 1 CNN layer with 1 attention layer, the implementation style before Attention Is All You Need paper. You can see in https://arxiv.org/abs/1703.03130

[hsd1503]

Here is a very beginning transformer1d repo https://github.com/hsd1503/transformer1d

[gerardsimons]

Thanks for the resources Hong, I will read up on that! What performance are you able to attain with this model? If you have time a table with all the different architectures and their performance on the Physionet data would be very useful! 🙏🏻