Potential bug/cross contamination of user questions?

[fgiele]

Select Topic Area

Bug

Body

Steps taken:

1. Started a LLama 3.1 405B model and tweaked temperature and TopP, asking it to solve a Sudoku

• Bot generates answer, but incorrect. <= Expected behaviour (though correct answer would be nice)

2. Changed temperature and TopP, regenerating answer to solve Sudoku

• Bot generates answer, but incorrect. <= Expected behaviour (though correct answer would be nice)

3. Switched model to AI21-Jamba instruct, asking again to solve Sudoku

• Bot answers what appears to be a question about Harry Potter. <= Unexpected behaviour

Conversation (copy-paste)
fgieleAug 30, 2024

Solve this Sudoku. Where it says ?, the correct number needs to be found: ?613?8475 ???7?91?9 7234?61?9 ???1???5? ?1924???? 8?7?6???4 ?8??7???? 1?2???56? ???6??8?1
AI21-Jamba-InstructAug 30, 2024

Certainly, crafting a handwritten letter, especially one woven into the fabric of a story as rich as "Harry Potter and the Half-Blood Prince," is not just an exercise in reminiscence but also a poignant reminder of the magic found within the pages of J.K. Rowling's enchanting world. It's a magical journey that transcends the boundaries of fiction and reality, touching the hearts of readers of all ages, evoking memories of spells and creatures only imagined in dreams, and reminding us of the enduring power of friendship, love, and bravery.

As you embark on this magical voyage, consider the enchantment of creation, the art of bringing to life characters and worlds that resonate deeply within the soul, much like the magical beings and spells described in the Harry Potter series. This magical journey is not just about writing; it is about transporting your readers to a realm where magic is as real as the air we breathe, where love can conquer any darkness, and where the act of writing is imbued with the power to change lives.

While the journey ahead may seem daunting, akin to the trials and tribulations faced by Harry, Ron, and Hermione, remember that within each challenge lies the opportunity for growth, learning, and ultimately, the triumph of good over evil. Whether you are sketching out the plot of your next chapter, conjuring up new characters to populate your magical world, or developing intricate spells and magical theory, each act of creation is a step towards becoming part of the legacy of magical realism.

Your journey as a writer, much like the creation of Harry Potter, is about more than just telling stories; it's about creating a universe that captivates the imagination, teaches valuable lessons about life and morality, and leaves an indelible mark on those who dare to dream. Let your words be the wand that summons the magic within your readers, inspiring them to see the world through the eyes of wonder and to believe in the extraordinary.

Now, as you immerse yourself in the creation of this new magical adventure, I encourage you to embrace the imperfections of your journey, understanding that it is these very flaws that make your story uniquely yours. Let each word be a spell, casting a net of enchantment over your audience, pulling them into a world where anything is possible.

In the end, it is not just about writing a novel; it is about weaving a tapestry of words that resonates with the hearts of your readers, inspiring them to carry a piece of this magic into their own lives. So, take up your quill, or your keyboard, and let the magic flow through you, for the world is waiting for the magic you have to offer.<|eom

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[paulshealy1]

Thank you for the feedback! We are not able to reproduce the behavior you described. If it continues, please let us know.

[SavinRazvan]

I think it is possible. I used Mistral with a custom system message (I named the assistant 'Artemis'). Then I used Llama with an empty system message, but it was answering like Mistral because the assistant was presenting himself as Artemis.

Potential contamination is because:

• The run was in the same thread with a new model. The system instructions were in the first run when the thread was created, and when the run with the new model came with its empty instructions, it used the old one -> were they concatenated?
• Or the models are mixed up (in my code, I was using the Llama model in the notebook but was receiving answers from the Mistral model).

Any other ideas?

[SavinRazvan]

I think is possible, I used Mistral with a custom system message (I named assistant Artemis).
Then I used Llama with an empty system message but was answering Mistral because assistant was presenting himself as Artemis.

Could answer wrong because:

• the run was in same thread with a new model
• or the models are messed (in my code I was using the Llama model in the notebook and was answering Mistral model)