Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI Chat: Remove Claude Instant model in favor of Claude Haiku and Claude Sonnet #23799

Merged
merged 1 commit into from
May 28, 2024

Conversation

petemill
Copy link
Member

@petemill petemill commented May 23, 2024

Also removes version (date) from model name, since server uses latest configured.

Resolves brave/brave-browser#37988

Submitter Checklist:

  • I confirm that no security/privacy review is needed and no other type of reviews are needed, or that I have requested them
  • There is a ticket for my issue
  • Used Github auto-closing keywords in the PR description above
  • Wrote a good PR/commit description
  • Squashed any review feedback or "fixup" commits before merge, so that history is a record of what happened in the repo, not your PR
  • Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
  • Checked the PR locally:
    • npm run test -- brave_browser_tests, npm run test -- brave_unit_tests wiki
    • npm run presubmit wiki, npm run gn_check, npm run tslint
  • Ran git rebase master (if needed)

Reviewer Checklist:

  • A security review is not needed, or a link to one is included in the PR description
  • New files have MPL-2.0 license header
  • Adequate test coverage exists to prevent regressions
  • Major classes, functions and non-trivial code blocks are well-commented
  • Changes in component dependencies are properly reflected in gn
  • Code follows the style guide
  • Test plan is specified in PR before merging

After-merge Checklist:

Test Plan:

  • Not premium user
  • Using a version before this commit, set Claude Instant to be the default
  • Verify every new conversation uses Claude Instant
  • Quit the browser and open again using this commit, with the same profile
  • Verify that every new conversation uses Claude Haiku
  • Repeat for premium user except every new conversation should now use Claude Sonnet

@petemill petemill self-assigned this May 23, 2024
@petemill petemill force-pushed the ai-chat-claude-instant-obsoletion branch from 5587978 to 23d19f3 Compare May 23, 2024 04:51
// Unretained is ok as credential manager is owned by this class,
// and it owns the mojo binding that is used to make async call in
// |GetPremiumStatus|.
base::Unretained(this)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reported by reviewdog 🐶
[semgrep] base::Unretained is most of the time unrequited, and a weak reference is better suited for secure coding.
Consider swapping Unretained for a weak reference.
base::Unretained usage may be acceptable when a callback owner is guaranteed
to be destroyed with the object base::Unretained is pointing to, for example:

- PrefChangeRegistrar
- base::*Timer
- mojo::Receiver
- any other class member destroyed when the class is deallocated


Source: https://github.com/brave/security-action/blob/main/assets/semgrep_rules/client/chromium-uaf.yaml


Cc @thypon @goodov @iefremov

@LorenzoMinto
Copy link
Member

Should we also address brave/brave-browser#38071 and brave/brave-browser#38195 in this PR? Should be a minimal addition.

@petemill petemill force-pushed the ai-chat-claude-instant-obsoletion branch 2 times, most recently from 00fcd8a to 0781484 Compare May 24, 2024 00:06
@petemill petemill force-pushed the ai-chat-claude-instant-obsoletion branch from 0781484 to de61f08 Compare May 24, 2024 04:15
@petemill
Copy link
Member Author

Should we also address brave/brave-browser#38071 and brave/brave-browser#38195 in this PR? Should be a minimal addition.

The server-side doesn't seem ready for that, so I'd prefer to keep them separate. Also the QA work for this is larger, so would like to allow them to get started verifying instead of waiting.

@petemill petemill merged commit a6c81fb into master May 28, 2024
19 checks passed
@petemill petemill deleted the ai-chat-claude-instant-obsoletion branch May 28, 2024 16:00
@github-actions github-actions bot added this to the 1.68.x - Nightly milestone May 28, 2024
brave-builds added a commit that referenced this pull request May 30, 2024
@stephendonner
Copy link
Contributor

stephendonner commented May 31, 2024

Verification PASSED using

Brave | 1.68.64 Chromium: 126.0.6478.26 (Official Build) nightly (64-bit)
-- | --
Revision | d9689f56dafd3c9d8622461f6b4864053b0706aa
OS | Windows 10 Version 22H2 (Build 19045.4474)

New profiles, default (Mixtral) - PASSED

Steps:

  1. install 1.68.64
  2. launch Brave using --enable-features=AIChat:conversation_api\true --env-leo=staging --env-ai-chat.bsg=dev --env-ai-chat-premium.bsg=dev
  3. opened brave://settings/leo-assistant
  4. confirmed Mixtral is default
  5. confirmed also-available Claude 3-family and Llama 3 8b models, below
  6. loaded https://people.com/angelina-jolie-and-brad-pitt-s-daughter-shiloh-18-files-to-drop-pitt-from-surname-8656440
  7. clicked on Summarize this page
  8. clicked on I understand
  9. waited for output

Confirmed Mixtral summarized the page content

example example
image image

Upgrades - PASSED

Case 1: Non-Premium user (Claude 3 Haiku) - PASSED

Steps:

  1. installed 1.66.118
  2. launched Brave using --enable-features=AIChat:conversation_api\true --env-leo=staging --env-ai-chat.bsg=dev --env-ai-chat-premium.bsg=dev
  3. opened brave://settings/leo-assistant
  4. changed Default model for new conversations from Mixtral to Claude Instant
  5. loaded https://en.wikipedia.org/wiki/Mozilla_Thunderbird
  6. clicked on Show Sidebar
  7. clicked on Leo
  8. clicked on Summarize this page
  9. NOTE: experienced a variant of Google doc summary is not rendered with Mixtral model when not opted in yet brave-browser#36086 😭
  10. quit Brave
  11. upgraded to 1.68.64, keeping profile intact, and renaming to Brave-Browser-Nightly
  12. launched Brave
  13. opened brave://settings/leo-assistant
  14. confirmed Claude 3 Haiku is now shown as Default model for new conversations
  15. clicked Summarize this page

Confirmed Claude 3 Haiku successfully summarized the Wikipedia page and supplanted Claude Instant, which is removed

example example example example
image image image image

Case 2: Premium user (Claude 3 Sonnet) - PASSED

Steps:

  1. installed 1.66.118
  2. launched Brave using --enable-features=AIChat:conversation_api\true --env-leo=staging --env-ai-chat.bsg=dev --env-ai-chat-premium.bsg=dev
  3. purchased and loaded Leo Premium credentials from account.bravesoftware.com using [email protected]
  4. set Default model for new conversations to Claude Instant in brave://settings/leo-assistant
  5. loaded https://www.reuters.com/world/us/white-house-race-plunges-into-uncharted-territory-trump-awaits-july-sentencing-2024-05-31/
  6. clicked on Summarize this page
  7. clicked I understand
  8. NOTE: experienced a variant of Google doc summary is not rendered with Mixtral model when not opted in yet brave-browser#36086 😭
  9. quit Brave
  10. renamed Brave-Browser profile to Brave-Browser-Nightly
  11. installed 1.68.64
  12. launched Brave using --enable-logging=stderr --v=2 --enable-features=AIChat:conversation_api\true --env-leo=staging --env-ai-chat.bsg=dev --env-ai-chat-premium.bsg=dev
  13. opened brave://settings/leo-assistant
  14. confirmed Claude Sonnet was the chosen default
  15. returned to https://www.reuters.com/world/us/white-house-race-plunges-into-uncharted-territory-trump-awaits-july-sentencing-2024-05-31/
  16. clicked on Summarize this page

Confirmed new default of Premium Claude 3 Sonnet replaced Claude Instant, and provided an accurate summarization

example example example example example
image image image image image

kjozwiak pushed a commit that referenced this pull request May 31, 2024
…ude Sonnet (uplift to 1.67.x) (#23919)

Uplift of #23799 (squashed) to beta
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Brave Leo]: Deprecate Claude Instant model
7 participants