Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix large number search in Postgres databases #22205

Closed
wants to merge 0 commits into from

Conversation

hanneskuettner
Copy link
Contributor

@hanneskuettner hanneskuettner commented Apr 16, 2024

Scope

What's changed:

  • Introduced a DB helper that wraps the intended changes for the Postgres client only and defaults for all others
  • Blackbox test need to be added, but the current state of blackbox tests does not allow that, so instead I manually tested the change and verified that the issue does not exist in other DB vendors.

Potential Risks / Drawbacks

None

Review Notes / Questions

  • Since this is the first time for me touching on DB helpers, is this one implemented in the correct spirit?
  • I also added a unit test case to specifically single out the helper usage in the applySearch function, I am unsure if that is the right location for it or if this should be dropped?

Fixes #17946

Copy link

changeset-bot bot commented Apr 16, 2024

⚠️ No Changeset found

Latest commit: 160b982

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@hanneskuettner
Copy link
Contributor Author

hanneskuettner commented Apr 16, 2024

Since blackbox tests aren't really in a working state right now I've tested searching large numbers with every database provider.

DB State Notes
MySQL
MariaDB
MS SQL Not running on my M3 MacBook
OracleDB Wasn't able to run it
CockroachDB
Postgres
RedShift Don't have access to any instance

I would appreciate hints on running the oracle docker we have in the development compose file as it fails with some networking issues on my end.

@hanneskuettner hanneskuettner changed the title [WIP] Fix large number search in Postgres databases Fix large number search in Postgres databases Apr 17, 2024
@hanneskuettner hanneskuettner marked this pull request as ready for review April 17, 2024 10:38
Copy link
Member

@br41nslug br41nslug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oracle and Postgres work as expected ✅
MS SQL still seems to have issues around bigInts and searching in my tests, will dive a bit deeper as you had issues getting that one running

Error: select top (@p0) [test_2].[id], [test_2].[big_int], [test_2].[int], [test_2].[string] from [test_2] where (([test_2].[id] = @p1) or ([test_2].[big_int] = @p2) or ([test_2].[int] = @p3) or LOWER([test_2].[string]) LIKE @P4) order by [test_2].[id] asc - Bigint must be safe integer or must be passed as string, saw 9223372036854775000

select top (@p0) [test_2].[id], [test_2].[big_int], [test_2].[int], [test_2].[string] from [test_2] where (([test_2].[id] = @p1) or ([test_2].[big_int] = @p2) or ([test_2].[int] = @p3) or LOWER([test_2].[string]) LIKE @P4) order by [test_2].[id] asc - The conversion of the nvarchar value '9223372036854775000' overflowed an int column.

Comment on lines 16 to 17
// maybe call this "addSearchCondition"?
orWhere(dbQuery: Knex.QueryBuilder, collection: string, name: string, value: NumericValue): Knex.QueryBuilder {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since orWhere is pretty generic what do you think of something like addSearchCondition

api/src/utils/apply-query.ts Outdated Show resolved Hide resolved
@@ -844,11 +858,25 @@ export async function applySearch(
});
}

function validateNumber(value: string, parsed: number) {
if (isNaN(parsed) || !Number.isFinite(parsed)) return false;
// move to its own util
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder to myself or hannes to move this to a util file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

Search error
2 participants