Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use token limits in core memory instead of character limits #2081

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

cpacker
Copy link
Collaborator

@cpacker cpacker commented Nov 21, 2024

2000 tokens is a good starting point? (~8k chars)

Pending design decision:

  • Do we make the Block.tokenizer_model nullable / Optional[str], or str with a default imported from constants.py?
    • For example, if the field is nullable, in the Pydantic validator that throws an error if the token count is higher than the limit, we can just use constants.DEFAULT_TOKENIZER_MODEL if self.tokenizer_model is None
    • Or (if we set defaults), self.tokenizer_model will be constants.DEFAULT_TOKENIZER_MODEL
    • Seems to me like the key difference is if we make the field nullable / have the default lazily inferred, then we can change the default for all existing blocks with new package versions?
  • Or do we put tokenizer_model inside of metadata_?

TODOs for later PR:

  • Initialize the tokenizer_model in the Memory class to be drawn from agent.llm_config.model
  • Make sure that when the agent.llm_config.model is changed, the memory.tokenizer_model is changed too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant