Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prompt refinement #227

Merged
merged 20 commits into from
Apr 23, 2024
Merged

Prompt refinement #227

merged 20 commits into from
Apr 23, 2024

Conversation

Swiftyos
Copy link
Contributor

No description provided.

@Swiftyos Swiftyos requested review from Torantulino and ntindle April 19, 2024 13:14
@Torantulino
Copy link
Member

Torantulino commented Apr 19, 2024

Looks potentially positive, but it's very hard to say with prompts. There's not much I can do to review this.

How can we objectively know whether it's better?

It would be very helpful to see the evidence you have so far @Swiftyos.

@Swiftyos Swiftyos requested a review from Torantulino April 19, 2024 13:55
@ntindle
Copy link
Member

ntindle commented Apr 19, 2024

add to the prisma parser tests the example here

@Swiftyos Swiftyos marked this pull request as draft April 19, 2024 14:14
@Swiftyos
Copy link
Contributor Author

Converted to draft until I have a good before and after benchmark.

@majdyz @ntindle @Torantulino @aarushik93 Please all feel free to commit to this PR any tweaks to the prompts (or target a pr into this one)

I will update the benchmark system then try and test if the prompts are improving or not

Base automatically changed from req-pr-comments to main April 22, 2024 08:32
@Swiftyos Swiftyos marked this pull request as ready for review April 23, 2024 08:49
@Swiftyos Swiftyos merged commit 7657759 into main Apr 23, 2024
3 checks passed
@Swiftyos Swiftyos deleted the prompt-refinement branch April 23, 2024 09:05
@Torantulino
Copy link
Member

Torantulino commented Apr 23, 2024

For the record:
I approved this merge, the results should be better in reality, but it's quicker for us to find out for ourselves than wait for the new system to be ready which will give us objective evidence.

Once this system is ready, we'll go through many iterative rounds of prompt refinement based on what we can measure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants