-
Notifications
You must be signed in to change notification settings - Fork 250
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SpanProto evaluation from checkpoint doesn't reproduce the f1 score #306
Comments
@wjn1996 would u check this pls |
any update on this? |
What is your recall?always 1? |
Hi Team SpanProto,
As described in the README file of span-proto directory, I trained with few-nerd 5way 5shot inter dataset. The prediction result shows ~0.82 f1 score as presented in the paper. However, when I am trying to use the trained checkpoint for prediction only i.e. adding --resume_from_checkpoint=./outputs/bert-base-uncased or --resume_from_checkpoint=./outputs/bert-base-uncased/checkpoint-7000 etc and remove --do_train argument, the result is bad like ~0.1 f1 score. Could you please tell me how can I reproduce the result of the test set using the trained checkpoint once the training is complete?
Thanks,
sayef
The text was updated successfully, but these errors were encountered: