Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

โœจ feat: add custom stream handle support for LobeOpenAICompatibleFactory #5039

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

hezhijie0327
Copy link
Contributor

@hezhijie0327 hezhijie0327 commented Dec 16, 2024

๐Ÿ’ป ๅ˜ๆ›ด็ฑปๅž‹ | Change Type

  • โœจ feat
  • ๐Ÿ› fix
  • โ™ป๏ธ refactor
  • ๐Ÿ’„ style
  • ๐Ÿ‘ท build
  • โšก๏ธ perf
  • ๐Ÿ“ docs
  • ๐Ÿ”จ chore

๐Ÿ”€ ๅ˜ๆ›ด่ฏดๆ˜Ž | Description of Change

  1. ไธบ LobeOpenAICompatibleFactory ๅผ•ๅ…ฅ handleStream ๅŠ handleTransformResponseToStream ๆŽฅๅฃ๏ผŒ็”จไบŽๅบ”็”จ่‡ชๅฎšไน‰ๆตๅค„็†
  2. Qwen ๅˆ‡ๅ›ž LobeOpenAICompatibleFactory
    a. ๅฏน่ฏญ่จ€ๆจกๅž‹้ป˜่ฎคๅฏ็”จๅ†…็ฝฎ่”็ฝ‘ๆœ็ดข่ƒฝๅŠ›๏ผŒๅผ•ๅ…ฅ enable_search (ๅฐ่ฑก้‡Œไน‹ๅ‰ๆœ‰ไธช Issue ๆไบ†่ฟ™ไธช feat๏ผŒ่ฟ‡ไผšๅ„ฟ็ฟปไธ‹) [Request] ๅขžๅŠ ้€šไน‰ๅƒ้—ฎๆ‰“ๅผ€่”็ฝ‘ๆœ็ดข็š„ๅผ€ๅ…ณย #4877
    b. ๅฏน frequency_penalty presence_penalty temperature top_p ่ฟ›่กŒ่Œƒๅ›ด้™ๅˆถ
    - ref: https://help.aliyun.com/zh/dashscope/developer-reference/tongyi-qianwen-vl-plus-api
    - ref: https://help.aliyun.com/zh/dashscope/developer-reference/qwen-api
ๅŠŸ่ƒฝ ๆ•ˆๆžœ
Qwen ไบ’่”็ฝ‘ๆœ็ดข imageimage
ๅคšๆจกๆ€ image
ๅทฅๅ…ท่ฐƒ็”จ image
  1. ่กฅ้ฝไน‹ๅ‰ Spark ็ผบๅฐ‘็š„ function call ็š„ๅฎž็Žฐ
    a. ้€š่ฟ‡ๆ”น้€ ๅฎž็Žฐไบ† Spark ็š„ Function Call ๅŠŸ่ƒฝ๏ผˆๆ•ดไฝ“ๅทฅๅ…ท่ฐƒ็”จๆ•ˆๆžœไธๆ˜ฏ็‰นๅˆซๅฅฝ๏ผŒๅญ˜ๅœจ้™ๅˆถ๏ผ‰
    b. ๆ”ฏๆŒๆตๅผๅŠ้žๆตๅผๅฏน่ฏ
    c. ่ฏ•ไบ†ไธ‹ๅทฅๅ…ท่ฐƒ็”จๆ”ฏๆŒๆตๅผ๏ผŒไฝ†ๆ˜ฏ้™คไบ† max-32k ๆจกๅž‹ๆ”ฏๆŒๅทฅๅ…ทๅ›ž่ฐƒๅค–๏ผŒๅ…ถไฝ™ๅ‡ไธๆ”ฏๆŒ๏ผˆcurl ่ฐƒ็”จๆต‹่ฏ•ไนŸไธบ็ฉบ๏ผ›max-32k ๅญ˜ๅœจ็›ธๅŒ payload๏ผŒไธŠไธ€็ง’่ฟ˜่ƒฝๅ‡บๅ†…ๅฎน๏ผŒๅ†ๆฌกๆ‰ง่กŒ็›ดๆŽฅ่ฟ”ๅ›ž็ฉบ็š„้—ฎ้ข˜๏ผ‰
Function Name ๆ่ฟฐ
transformSparkStream ๅค„็†ๆตๅผ
transformSparkResponseToStream ๅค„็†้žๆตๅผ

Spark ็š„ๅทฅๅ…ท่ฐƒ็”จๆตๅพˆๆ€ช๏ผŒ่ฟ”ๅ›ž็š„ๆ˜ฏๅฏน่ฑกไธๆ˜ฏๆ•ฐ็ป„๏ผŒ้€š่ฟ‡้ญ”ๆ”น OpenAI ๆตๅค„็†ๅฏน้žๆ ‡ๅ‡†่ฟ”ๅ›žๅผบๅˆถ่ฝฌๆขไธบๆ•ฐ็ป„


PS: ็ป™่ฎฏ้ฃžๅผ€ไบ†ๅทฅๅ•๏ผŒๅฐ†่ฟ‘ไธ‰ไธชๅคšๆœˆๆ— ่ฟ›ๅฑ•
image

๐Ÿ“ ่กฅๅ……ไฟกๆฏ | Additional Information

ๆจกๅ— ๆ•ˆๆžœ
ๅฏน่ฏ image
ๅทฅๅ…ท่ฐƒ็”จ image

้žๆตๅผๅ“ๅบ”ๆต

{
    "code": 0,
    "message": "Success",
    "sid": "cha000b95d2@dx193cef49db1b894532",
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "",
                "tool_calls": {
                    "type": "function",
                    "function": {
                        "arguments": "{\"city\":\"Shanghai\"}",
                        "name": "realtime-weather____fetchCurrentWeather"
                    }
                }
            },
            "index": 0
        }
    ],
    "usage": {
        "prompt_tokens": 8,
        "completion_tokens": 0,
        "total_tokens": 8
    }
}

ๆตๅผ่พ“ๅ‡บๅ“ๅบ”ๆต

{
    "code": 0,
    "message": "Success",
    "sid": "cha000b6cd0@dx193cef628319a4b532",
    "id": "cha000b6cd0@dx193cef628319a4b532",
    "created": 1734344060,
    "choices": [
        {
            "delta": {
                "role": "assistant",
                "content": "",
                "tool_calls": {
                    "type": "function",
                    "function": {
                        "arguments": "{\"city\":\"Shanghai\"}",
                        "name": "realtime-weather____fetchCurrentWeather"
                    }
                }
            },
            "index": 0
        }
    ],
    "usage": {
        "prompt_tokens": 8,
        "completion_tokens": 0,
        "total_tokens": 8
    }
}

Copy link

vercel bot commented Dec 16, 2024

@hezhijie0327 is attempting to deploy a commit to the LobeHub Team on Vercel.

A member of the Team first needs to authorize it.

@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Dec 16, 2024
@lobehubbot
Copy link
Member

๐Ÿ‘ @hezhijie0327

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
้žๅธธๆ„Ÿ่ฐขๆ‚จๆๅ‡บๆ‹‰ๅ–่ฏทๆฑ‚ๅนถไธบๆˆ‘ไปฌ็š„็คพๅŒบๅšๅ‡บ่ดก็Œฎ๏ผŒ่ฏท็กฎไฟๆ‚จๅทฒ็ป้ตๅพชไบ†ๆˆ‘ไปฌ็š„่ดก็ŒฎๆŒ‡ๅ—๏ผŒๆˆ‘ไปฌไผšๅฐฝๅฟซๅฎกๆŸฅๅฎƒใ€‚
ๅฆ‚ๆžœๆ‚จ้‡ๅˆฐไปปไฝ•้—ฎ้ข˜๏ผŒ่ฏท้šๆ—ถไธŽๆˆ‘ไปฌ่”็ณปใ€‚

@dosubot dosubot bot added the ๐Ÿ’„ Design Design an style | ๆ ทๅผ้—ฎ้ข˜ label Dec 16, 2024
Copy link

codecov bot commented Dec 16, 2024

Codecov Report

Attention: Patch coverage is 87.19212% with 26 lines in your changes missing coverage. Please review.

Project coverage is 92.16%. Comparing base (50e0dc4) to head (cd6237e).

Files with missing lines Patch % Lines
src/libs/agent-runtime/utils/streams/spark.ts 82.83% 23 Missing โš ๏ธ
src/libs/agent-runtime/qwen/index.ts 93.61% 3 Missing โš ๏ธ
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5039      +/-   ##
==========================================
- Coverage   92.20%   92.16%   -0.05%     
==========================================
  Files         568      569       +1     
  Lines       43395    43463      +68     
  Branches     2706     3890    +1184     
==========================================
+ Hits        40012    40056      +44     
- Misses       3383     3407      +24     
Flag Coverage ฮ”
app 92.16% <87.19%> (-0.05%) โฌ‡๏ธ
server 97.66% <รธ> (รธ)

Flags with carried forward coverage won't be shown. Click here to find out more.

โ˜” View full report in Codecov by Sentry.
๐Ÿ“ข Have feedback on the report? Share it here.

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Dec 16, 2024
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

่ฟ™ๅ—ๆ˜ฏไธๆ˜ฏๆœ€ๅฅฝ็œ‹ไธ‹ๅ“ช้‡Œ้œ€่ฆ็ป™ๅทฅๅŽ‚ๅ‡ฝๆ•ฐๅผ€ๆงฝ็š„๏ผŒ็ป™ๅผ€ไธชๆงฝ๏ผŸๅ•็‹ฌๅฎž็Žฐไธ€ไธชๆ„Ÿ่ง‰ๅŽ็ปญ็š„็ปดๆŠคๆˆๆœฌๆฏ”่พƒ้ซ˜

Copy link
Contributor Author

@hezhijie0327 hezhijie0327 Dec 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ๅŠ ไบ† handleStream ๅŠ handleTtransformResponseToStream ๆŽฅๅฃ๏ผŒๅŠŸ่ƒฝไธŠๆฒกๅ•ฅ้—ฎ้ข˜ไบ† ๏ผŒ่ฟ˜ๅทฎไธ€ไธช้žๆตๅผ่ฝฌๆตๅผ็š„่‡ชๅฎšไน‰ๅค„็†

@hezhijie0327 hezhijie0327 changed the title ๐Ÿ’„ style: add function call support for Spark โœจ feat: add custom stream handle support for LobeOpenAICompatibleFactory Dec 21, 2024
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Dec 21, 2024
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Dec 21, 2024
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Dec 21, 2024
@hezhijie0327
Copy link
Contributor Author

@arvinxx ๆ”น้€ ๅฅฝไบ†

  1. ้กบๅธฆๆŠŠ Qwen ไนŸๅˆ‡ๆขๅ›ž LobeOpenAICompatibleFactory ๅนถๅผ•ๅ…ฅ handleStream: QwenAIStream ๅค„็†่‡ชๅฎšไน‰ๆต
  2. ไธบ Qwen Chat ๆจกๅž‹ๅผ•ๅ…ฅ enable_search ้ป˜่ฎคๅฏ็”จๅ†…ๅปบ่”็ฝ‘ๅŠŸ่ƒฝ
  3. ไธบ Qwen ๅผ•ๅ…ฅ QwenLegacyModels ๆ•ฐ็ป„๏ผŒ็”จไบŽๅค„็†้ƒจๅˆ†ๆจกๅž‹ไธๆ”ฏๆŒ presence_penalty

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. ๐Ÿ‘ฏ๐Ÿ‘ญ๐Ÿป๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘๐Ÿ‘ซ๐Ÿง‘๐Ÿฟโ€๐Ÿคโ€๐Ÿง‘๐Ÿป๐Ÿ‘ฉ๐Ÿพโ€๐Ÿคโ€๐Ÿ‘จ๐Ÿฟ๐Ÿ‘ฌ๐Ÿฟ


@arvinxx has been transformed

  1. By the way, switch Qwen back to LobeOpenAICompatibleFactory and introduce handleStream: QwenAIStream to process custom streams
  2. Introduce enable_search to the Qwen Chat model to enable the built-in networking function by default
  3. Introduce the QwenLegacyModels array to Qwen to handle some models that do not support presence_penalty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
๐Ÿ’„ Design Design an style | ๆ ทๅผ้—ฎ้ข˜ size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants