Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use JSON field for concat field #4924

Open
PSeitz opened this issue Apr 29, 2024 · 0 comments · May be fixed by #4937
Open

use JSON field for concat field #4924

PSeitz opened this issue Apr 29, 2024 · 0 comments · May be fixed by #4937
Labels
enhancement New feature or request tech debt

Comments

@PSeitz
Copy link
Contributor

PSeitz commented Apr 29, 2024

The concat fields introduced in this PR #4773 uses a multi-valued text field to concatenate the data into one field. Non-string types are converted and indexed as string type.

We could use the JSON field instead of a multivalued text field for the concat field.
That way we keep the type info and there are some synergies since JSON has the same problem space when indexing and querying data with mixed types.

e.g. how would we know if the user input in a query is a text, number, date, etc. and how to generate queries that can handle different types with potential ambiguous cases.
On the indexing side it is similar, that we may encounter e.g. "2024-04-28T17:31:59.893Z" and should convert that to a date.

The change should be straightforward: Change the field type to JSON and remove the conversion logic.

Impact

Impact should be minimal, since the feature is pretty new and not yet released.
But may be some customers are already using it. @trinity-1686a @fmassot
The field type would change, but existing usage should still work, since we check the fields type anyways?

Update: There is one user using it, and it's okay to break it (which I think doesn't even happen).

@PSeitz PSeitz added enhancement New feature or request tech debt labels Apr 29, 2024
PSeitz added a commit that referenced this issue May 6, 2024
use JSON for concat fields instead of concatenating text fields.

With quickwit-oss/tantivy#2383 we support now
non-object values on the root in the JSON field.
As a nice side-effect, this will make regular JSON fields more powerful.
Instead only for nested types, JSON is now also useable for flat mixed-type fields.

Closes: #4924
@PSeitz PSeitz linked a pull request May 6, 2024 that will close this issue
PSeitz added a commit that referenced this issue May 6, 2024
use JSON for concat fields instead of concatenating text fields.

With quickwit-oss/tantivy#2383 we support now
non-object values on the root in the JSON field.
As a nice side-effect, this will make regular JSON fields more powerful.
Instead only for nested types, JSON is now also useable for flat mixed-type fields.

Closes: #4924
PSeitz added a commit that referenced this issue May 6, 2024
use JSON for concat fields instead of concatenating text fields.

With quickwit-oss/tantivy#2383 we support now
non-object values on the root in the JSON field.
As a nice side-effect, this will make regular JSON fields more powerful.
Instead only for nested types, JSON is now also useable for flat mixed-type fields.

Closes: #4924
PSeitz added a commit that referenced this issue May 6, 2024
use JSON for concat fields instead of concatenating text fields.

With quickwit-oss/tantivy#2383 we support now
non-object values on the root in the JSON field.
As a nice side-effect, this will make regular JSON fields more powerful.
Instead only for nested types, JSON is now also useable for flat mixed-type fields.

Closes: #4924
PSeitz added a commit that referenced this issue May 13, 2024
use JSON for concat fields instead of concatenating text fields.

With quickwit-oss/tantivy#2383 we support now
non-object values on the root in the JSON field.
As a nice side-effect, this will make regular JSON fields more powerful.
Instead only for nested types, JSON is now also useable for flat mixed-type fields.

Closes: #4924
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request tech debt
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant