Support for Mistral Inference

[michaelgruner]

Hey @dusty-nv, something like 10 days ago Mistral released Mistral-7B-Instruct-v0.3. What is interesting about it is that it is (to the best of my knowledge) the first open source model to support native function calling. Not only they've fine tuned the model to do function calling, but the tokenizer was also modified and there are special function calling tokens, and so on. To take advantage of this we need to use mistral-inference instead of transformers.

I thought this would be more robust for the home assistant than plain prompting. I think NanoLLM is a better repo for this, but it's definitely more work. I'll be working on this over the weekend.

[dusty-nv]

Thanks @michaelgruner , yes these functioned-tuned models are great looks like they follow OpenAI tool spec too, will have to look into this more as to what chat template the actual model follows. MLC supports Mistral and I know people were running the Mixtral-8x7B with it, will have to try this sometime.

And yes NanoLLM and/or Jetson AI Lab would probably be where this lands as it doesn't actually require a new container. We should have a page for the models that can do function calling and how to use them.