Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure generating response #84

Open
Emmanuel-Develops opened this issue May 14, 2024 · 2 comments
Open

Failure generating response #84

Emmanuel-Develops opened this issue May 14, 2024 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@Emmanuel-Develops
Copy link
Collaborator

problem

I'm not able to get an answer for "silent payments" but other queries work
Screenshot_20240514_032159_Samsung_InternetScreenshot 2024-05-14 at 12 45 25 PM

deductions

  • cache?: second image shows no cached answers were used
  • sources?: second image shows only one source was used.
    Possible explanation: Token limit cuts off the remaining sources as they were too large
@kouloumos
Copy link
Contributor

kouloumos commented May 20, 2024

Possible explanation: Token limit cuts off the remaining sources as they were too large

Token limit is 7000 characters. The source that is included (github) is ~273 tokens based on this tool.

edit: Okay, my bad, so what you are saying is that the remaining sources add up to more than 7000 tokens. Do we have access to what those sources are? So that we can verify it?

@Emmanuel-Develops
Copy link
Collaborator Author

Emmanuel-Develops commented May 20, 2024

edit: Okay, my bad, so what you are saying is that the remaining sources add up to more than 7000 tokens. Do we have access to what those sources are? So that we can verify it?

Yeah, but more like the next relevant source in addition with the most relevant (the single source shown) adds up to more than 7000 tokens.

Yeah, on local you can disable the tokenEnforcer and see the sources used or log the sources before then. I ran the query again and these are the results with tokenEnforcer disabled.
Note: the order of the sources may not be exactly what the user got the first time

[0]: https://bitcoinops.org/en/topics/silent-payments
[1]: https://btctranscripts.com/bitcoin-explained/silent-payments
[2]: https://btctranscripts.com/bitcoin-core-dev-tech/2023-04/2023-04-26-silent-payments
[3]: https://btctranscripts.com/bitcoin-design/learning-bitcoin-and-design/silent-payments
[4]: https://delvingbitcoin.org/t/about-the-wg-silent-payments-category/876
[5]: https://btctranscripts.com/bitcoin-explained/accounts-for-bitcoin-easypaysy
[6]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020519.html
[7]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021844.html
[8]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-August/020885.html
[9]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021864.html

source [1] token count is over 11,000.

With token enforcer you have this (this is from the same prompt)

{ tokenLength: 39392 }
{ tokenLength: 36922 }
{ tokenLength: 36616 }
{ tokenLength: 32113 }
{ tokenLength: 31568 }
{ tokenLength: 24684 }
{ tokenLength: 24558 }
{ tokenLength: 11179 }
{ tokenLength: 10278 }
{ tokenLength: 299 }
enforceToken: 779.628ms

so it keeps removing sources from least relevant till it fits into the context window.
Therefore explains the single source [0] that was selected.

@aassoiants aassoiants moved this to 📋 Backlog in ChatBTC Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 🆕 Todo
Development

No branches or pull requests

3 participants