attention-is-all-you-need

My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.

python deep-learning jupyter transformers pytorch transformer attention deeplearning attention-mechanism attention-is-all-you-need pytorch-transformers pytorch-transformer transformer-tutorial original-transformer

Updated Dec 27, 2020
Jupyter Notebook

Separius / awesome-fast-attention

Star

list of efficient attention modules

awesome transformer attention attention-is-all-you-need multihead-attention reformer self-attention transformer-network longformer linformer

Updated Aug 23, 2021
Python

brightmart / bert_language_understanding

Star

Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN

nlp text-classification question-answering document-classification transfer-learning fasttext language-model textcnn attention-is-all-you-need self-attention transformer-encoder bert-model pre-training language-understanding

Updated Jan 1, 2019
Python

kaituoxu / Speech-Transformer

Star

A PyTorch implementation of Speech Transformer, an End-to-End ASR with Transformer network on Mandarin Chinese.

end-to-end pytorch transformer attention asr attention-is-all-you-need self-attention

Updated Apr 6, 2023
Python

lsdefine / attention-is-all-you-need-keras

Star

A Keras+TensorFlow Implementation of the Transformer: Attention Is All You Need

deep-learning keras keras-tensorflow attention-is-all-you-need attention-seq2seq

Updated Sep 24, 2021
Python

kyegomez / LongNet

Sponsor

Star

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

machine-learning artificial-intelligence transformer attention attention-is-all-you-need attention-mechanisms gpt3 gpt4 chatgpt context-length

Updated Jan 7, 2024
Python

hkproj / pytorch-transformer

Star

Attention is all you need implementation

pytorch transformer paper-implementations attention-is-all-you-need

Updated Jun 8, 2024
Jupyter Notebook

sooftware / kospeech

Sponsor

Star

Open-Source Toolkit for End-to-End Korean Automatic Speech Recognition leveraging PyTorch and Hydra.

end-to-end pytorch transformer speech-recognition las seq2seq jasper asr conformer attention-is-all-you-need korean-speech e2e-asr las-models ksponspeech

Updated May 27, 2023
Python

FreedomIntelligence / TextClassificationBenchmark

Star

A Benchmark of Text Classification in PyTorch

benchmark text-classification quantum cnn pytorch lstm rcnn attention-is-all-you-need crnn cnn-classification lstm-sentiment-analysis capusle

Updated Apr 20, 2024
Python

jayparks / transformer

Star

A Pytorch Implementation of "Attention is All You Need" and "Weighted Transformer Network for Machine Translation"

machine-translation attention-mechanism attention-is-all-you-need self-attention

Updated Oct 1, 2020
Python

lvapeab / nmt-keras

Star

Neural Machine Translation with Keras

machine-learning theano deep-learning tensorflow machine-translation keras transformer gru neural-machine-translation sequence-to-sequence nmt attention-mechanism web-demo attention-model lstm-networks attention-is-all-you-need attention-seq2seq

Updated Jul 30, 2021
Python

feifeibear / long-context-attention

Star

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

pytorch attention-is-all-you-need llm-training llm-inference ring-attention deepspeed-ulysses

Updated Nov 19, 2024
Python

kyegomez / CM3Leon

Sponsor

Star

An open source implementation of "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning", an all-new multi modal AI that uses just a decoder to generate both text and images

attention multimodality attention-is-all-you-need multimodal-learning multimodal imagegeneration dalle