Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Byte Pair Encoding (NLP) #196

Open
KushGrandhi opened this issue Nov 25, 2024 · 2 comments
Open

Byte Pair Encoding (NLP) #196

KushGrandhi opened this issue Nov 25, 2024 · 2 comments
Assignees

Comments

@KushGrandhi
Copy link

Byte Pair Encoding (BPE) for Subword Tokenization

Problem Statement:
Design and implement an algorithm to tokenize a given corpus into subword units based on the frequency of adjacent character pairs using the Byte Pair Encoding (BPE) method. The algorithm should iteratively merge the most frequent character pairs in the corpus to create a compact and meaningful token vocabulary.

The task is to implement BPE by:

  • Iteratively finding and merging the most frequent pairs of adjacent characters or subwords.
  • Updating the corpus and token vocabulary after each merge.
  • Continuing the process until a stopping criterion is reached (e.g., a fixed number of merges or achieving a desired vocabulary size).
@moe18
Copy link
Collaborator

moe18 commented Nov 25, 2024

Sounds like a good addition

@KushGrandhi
Copy link
Author

sharing this just for reference #199

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants