Jetty threads starvation #42139
Comments
|
You need to tell us exactly what you do to use those threads so we can reproduce. Btw: why do you have that c3p0 config? |
We are not doing anything special - several datasources are connected to the Metabase (Postgres, Oracle), from which we take information for the dashboards that users work with. idleConnectionTestPeriod was needed because network devices between the DBMS and Metabase sometimes unexpectedly broke network connections; maxIdleTime and unreturnedConnectionTimeout were added in an attempt to help with increasing number of threads. |
|
I also have dumps for different days - you can see the dynamics of increasing number of HelperThreads (Samurai, for example, can visualize the difference between several dumps): |
|
Can you upgrade to our latest 49.8 please Also when you say several datasources are connected to the Metabase can you quantify how many and what is the setting of the sync and scan for such Databases? |
Sure, will try it in a couple of days.
Only one Oracle datasource is most actively used - hourly sync, scan disabled. Other data sources (PostgreSQL - 4, Oracle - 4) - hourly sync, daily scan. |
|
So you have 8 scans that happen every day right? Can you check how long those take? You can check the logs for FINISHED keywords and for the update-field-values 
|
|
Looks like I forgot some datasources. Here are the logs for the last 5 days: |
Hello. Sorry for the delay - updated to 49.10 this morning. We'll take a look at Jetty in a couple of days and I'll come back with information. |
Describe the bug
Hello.
I encountered the following problem in Metabase - during operation the number of busy jetty threads gradually increases, after which at some point the application stops responding to healthcheck probes:
At the same time, the number of HelperThreads in the thread dump increases. Here is an application dump with uptime for about a week:
metabase-20240502.dmp
Adding maxIdleTime and unreturnedConnectionTimeout parameters does not help. Updating the version from 0.45 to 0.48 also does not help.
Current configuration:
To Reproduce
Expected behavior
The number of busy Jetty threads does not increase over time.
Logs
2024-04-26 09:31:08,696 INFO metabase.core :: Starting Metabase version v0.48.8 (a900c85)
2024-04-27 00:00:55,212 DEBUG middleware.log :: GET /api/health 200 158.6 µs (0 DB calls) App DB connections: 0/10 Jetty threads: 9/100 (2 idle, 0 queued) (202 total active threads) Queries in flight: 1 (0 queued)
2024-04-29 00:01:35,991 DEBUG middleware.log :: GET /api/health 200 150.0 µs (0 DB calls) App DB connections: 0/8 Jetty threads: 39/100 (3 idle, 0 queued) (267 total active threads) Queries in flight: 0 (0 queued)
2024-05-02 00:02:25,991 DEBUG middleware.log :: GET /api/health 200 153.4 µs (0 DB calls) App DB connections: 0/15 Jetty threads: 44/100 (2 idle, 0 queued) (283 total active threads) Queries in flight: 0 (0 queued)
Information about your Metabase installation
{ "browser-info": { "language": "en-US", "platform": "MacIntel", "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0", "vendor": "" }, "system-info": { "file.encoding": "UTF-8", "java.runtime.name": "OpenJDK Runtime Environment", "java.runtime.version": "11.0.22+7", "java.vendor": "Eclipse Adoptium", "java.vendor.url": "https://adoptium.net/", "java.version": "11.0.22", "java.vm.name": "OpenJDK 64-Bit Server VM", "java.vm.version": "11.0.22+7", "os.name": "Linux", "os.version": "5.4.0-177-generic", "user.language": "en", "user.timezone": "GMT" }, "metabase-info": { "databases": [ "googleanalytics", "postgres", "bigquery-cloud-sdk", "oracle" ], "hosting-env": "unknown", "application-database": "postgres", "application-database-details": { "database": { "name": "PostgreSQL", "version": "14.11 (Ubuntu 14.11-1.pgdg22.04+1)" }, "jdbc-driver": { "name": "PostgreSQL JDBC Driver", "version": "42.6.0" } }, "run-mode": "prod", "version": { "date": "2024-03-04", "tag": "v0.48.8", "hash": "a900c85" }, "settings": { "report-timezone": "Europe/Riga" } } }Severity
annoying
Additional context
No response
The text was updated successfully, but these errors were encountered: