Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

closedqa prompt is not adequate for gpt-4-0613 #1228

Open
JasonGross opened this issue Jun 24, 2023 · 0 comments · May be fixed by #1258
Open

closedqa prompt is not adequate for gpt-4-0613 #1228

JasonGross opened this issue Jun 24, 2023 · 0 comments · May be fixed by #1258

Comments

@JasonGross
Copy link
Contributor

JasonGross commented Jun 24, 2023

It seems that GPT-4 neglects to follow the instructions in the closedqa prompt much more than gpt-3.5-turbo. See, for example, #1200 (comment) where gpt-4 gives 9 invalid responses out of 47, while gpt-3.5-turbo does not give any invalid responses. Does this hold across the other evals in the repo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant