-
Notifications
You must be signed in to change notification settings - Fork 8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Dataset quality] Adjust degraded docs query #179227
Comments
Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs) |
Engineering effort wise, what is the least effort? I'm thinking of starting with 1 and then follow up with 3. Or if 3 is as simple as one, directly jump to 3. One thing we have to consider is also how much load we put on the cluster. I assume for the default of 24h it should be doable, but for longer periods, it could take a long time and make Elasticsearch read all the data. |
Thanks for your input @ruflin, I have now another question: is there a way to move those indices that don't support the aggregation to a state where they support it? |
For data streams, my understanding is as soon as a rollover is triggered and the new index is created, it will be available on all the new data. @salvatore-campagna Can you confirm? |
We could guide users to do this, so next time they can benefit fully from the aggregation |
@mdbirnstiehl could you help us here with some communications?
|
How about:
My concern is whether or not "supporting |
Just my 2c here but I'd go with option 1 and warn users about partial results and encourage them to roll over the data stream. I'd rather not give users a foot-gun that can affect their cluster's performance. If we choose to still do a fallback query for indices that don't support aggregations for _ignored, I'd implement it as an opt-in, i.e. the user needs to click a button to trigger the slow aggregation. |
Hey Felix, initially I though we would need to change the query to something like
With that type of query we will get failures and partial results, the thing is that query will only tell us how many documents we have containing a value of an specific
The problem with that type of response is that we are not able to tell how many documents in that dataset actually have Maybe I'm missing something but still I feel we would need to make use of |
I think I've found a simple solution to get the ratio of degraded documents only for indices that have doc_values for the _ignored field. Instead of doing an {
"size": 0,
"aggs": {
"non_degraded": {
"missing": {
"field": "_ignored"
}
}
}
} Based on some quick testing, the aggregation fails on shards where the field doesn't have doc_values but succeeds on other shards. Click to expand example
|
Thanks @felixbarny, I did some quick testings in For 5d, including non aggregatable indices
*missing is not returning complete data, instead it returns failures for the indices not supporting the For latest 24h, all indices supporting the aggregation
Avg in performance is very similar, do you think for higher amount of data we could conclude that using I was also trying to split the query because I think that will speed things up for most of the cases (I expect total of documents with ignored property to be a reduced number compared to the rest) if we could request the queries in parallel Queries
Doing that I got the following numbers For 5d, including non aggregatable indices
For latest 24h, all indices supporting the aggregation
If we could do that, parallelise, we might lowering the time taken by elastic search to the ~half because the critical path will be on the longest query. wdyt? |
Relates to #179227. After gathering some numbers around possible tweaks of the current degradedDocs query ([more information](#179227 (comment))), I decided to move forward and split the query to reduce the time taken by elastic search aggregating on data streams. This PR contains the following changes: - `mSearch` method was added to `DatasetQualityESClient` to allow the usage of [multi search](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html). - `degradedDocsRt` was changed to now include not only the amount of degradedDocs but also the total docs for the datastreams within the timerange selected Nothing visible has changed in terms of functionality https://github.com/elastic/kibana/assets/1313018/1d836446-7d35-4ff2-8356-16a8087ab505
…ation (#183183) Closes #179227. ## 📝 Summary This PR adds a warning to main page and flyout whenever a dataStream has indices that doesn't support `_ignored` aggregation ## 🎥 Demo https://github.com/elastic/kibana/assets/1313018/c6d5fd81-d9a9-4fcc-92d0-8e65b996df9c #### Flyout https://github.com/elastic/kibana/assets/1313018/2972e104-8b10-413a-a430-3efc5252b757
I think we should run the query only for indices that have doc values on |
@salvatore-campagna What you propose above adds a lot of complexity on the UI side for a "temporary" problem for our users. We opted to go for a banner on the top. |
📓 Summary
When elastic/elasticsearch#59946 lands we would need to improve the query to get degraded logs.
✔️ Acceptance criteria
_ignored
aggregation in the selected timeframe.❓ Open questions
_ignored
we let users know that this will take more time than expected and perform the query with the aggregation on the indices that support it and find an alternative (related to 2
) to get the information for indices that don't support the aggregationThe text was updated successfully, but these errors were encountered: