-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there hope Grit-Embedding beats this task? #22
Comments
This is very interesting! I think it should be possible to get there with pure embeddings. GritLM is pretty close - what instruction are you using? Maybe optimizing the instruction a bit gets the model there. If you have many tricky examples like this, then fine-tuning on them should help I think. However, if it's just generic hiring data, I'm not as sure. |
I tried 2 different instructions:
The second one performed better and the scores from the table are from the second one. |
Thank you for this great model and the corresponding paper. I will definitely cite you in my thesis :)
In the attached experiment, I am trying to "trick" the model by using lexically identical words in the document that is less desired.
The first run was passed by GritLM but not by Instructor.
However, on the second run, where I changed "advanced" to "in-depth" to create yet another lexical match, the model was finally tricked.
Can embeddings be strong enough to beat this test, or is understanding such nuances a task only cross-encoders can solve? As this is kinda recruitment-like scenario, do you think that additional fine-tuning, with some recruitment-domain dataset, would help?
The text was updated successfully, but these errors were encountered: