Cohere Command R Input Token Limit?

[ayan4m1]

Models

Question

Body

This model lists context as 131k input tokens.

Context
131k input · 4k output

Yet with the sample for making a single request, I get this error when passing ~125k tokens

The sample encountered an error: {
  code: 'tokens_limit_reached',
  message: 'Request body too large for cohere-command-r model. Max size: 8000 tokens.',
  details: null
}

Why do the two numbers (131k and 8k) not line up?

[matthewisabel]

Sorry for the confusion. Those are the characteristics of the model, although we apply token limits you can read here: https://docs.github.com/en/github-models/prototyping-with-ai-models#rate-limits

We're still evaluating these limits as this is limited preview and there isn't a seamless upgrade path yet to let you go beyond these constraints. Hopefully more updates soon and I'll think about how we can provide more clarity here.

[ayan4m1]

OK, I was wondering if there were then a way to potentially feed the tokens in 8k at a time, up to the context limit... not sure if there is a way to call /chat/completions as a "follow-up" or if each conversation needs to be a discrete entity.

Regardless, thanks for the quick response, that makes sense. I will wait for more info, thanks for providing this service!