Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grafana crashing #15

Open
knmorgan opened this issue Dec 20, 2022 · 1 comment
Open

Grafana crashing #15

knmorgan opened this issue Dec 20, 2022 · 1 comment

Comments

@knmorgan
Copy link

I had this successfully running on my Synology previously, but re-initialized the volume and cannot get it to work again. When following the directions on this repo I get the following logs from supervisord ad infinitum.

2022-12-20 04:11:10,233 INFO spawned: 'grafana' with pid 5185
2022-12-20 04:11:11,256 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-20 04:11:11,731 INFO exited: grafana (exit status 1; not expected)
2022-12-20 04:11:11,731 INFO reaped unknown pid 5196
2022-12-20 04:11:11,733 INFO spawned: 'grafana' with pid 5205
2022-12-20 04:11:12,756 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-20 04:11:12,757 INFO reaped unknown pid 5216
2022-12-20 04:11:12,767 INFO exited: grafana (exit status 1; not expected)

Looking in the grafana log I see complaints about the database being locked initially:

logger=settings t=2022-12-20T04:05:14.515925475Z level=info msg="Starting Grafana" version=9.1.3 commit=f3f9a818c2 branch=HEAD compiled=2022-09-05T15:59:46Z
logger=settings t=2022-12-20T04:05:14.516336991Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-12-20T04:05:14.516396709Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-12-20T04:05:14.516425871Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-12-20T04:05:14.516454471Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-12-20T04:05:14.516483262Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-12-20T04:05:14.516511072Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-12-20T04:05:14.516539953Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_TYPE=sqlite3"
logger=settings t=2022-12-20T04:05:14.516568518Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-12-20T04:05:14.516595898Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-12-20T04:05:14.516622801Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-12-20T04:05:14.516649685Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-12-20T04:05:14.51667711Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-12-20T04:05:14.516704603Z level=info msg="App mode production"
logger=sqlstore t=2022-12-20T04:05:14.516869675Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-12-20T04:05:14.516922141Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxr-xr-x expected=-rw-r-----
logger=migrator t=2022-12-20T04:05:15.021613642Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7"
logger=settings t=2022-12-20T04:05:15.574853239Z level=info msg="Starting Grafana" version=9.1.3 commit=f3f9a818c2 branch=HEAD compiled=2022-09-05T15:59:46Z
logger=settings t=2022-12-20T04:05:15.575354207Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-12-20T04:05:15.575429981Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-12-20T04:05:15.575478012Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-12-20T04:05:15.57550559Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-12-20T04:05:15.575532263Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-12-20T04:05:15.575559553Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-12-20T04:05:15.575587645Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_TYPE=sqlite3"
logger=settings t=2022-12-20T04:05:15.57561536Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-12-20T04:05:15.575641907Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-12-20T04:05:15.57566846Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-12-20T04:05:15.575694686Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-12-20T04:05:15.575721178Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-12-20T04:05:15.57574836Z level=info msg="App mode production"
logger=sqlstore t=2022-12-20T04:05:15.575924431Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-12-20T04:05:15.575976855Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxr-xr-x expected=-rw-r-----
logger=migrator t=2022-12-20T04:05:15.588282961Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7"
logger=migrator t=2022-12-20T04:05:16.419610267Z level=error msg="alert migration failure: could not get migration log" error="failed to check table existence: database is locked"
logger=migrator t=2022-12-20T04:05:16.548090565Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7"

This lasts about 5 minutes, after which it no longer complains, but gets stuck in a loop without any apparent errors:

logger=settings t=2022-12-20T04:15:12.661900194Z level=info msg="Starting Grafana" version=9.1.3 commit=f3f9a818c2 branch=HEAD compiled=2022-09-05T15:59:46Z
logger=settings t=2022-12-20T04:15:12.662414423Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-12-20T04:15:12.662468342Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-12-20T04:15:12.662495231Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-12-20T04:15:12.662522201Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-12-20T04:15:12.662551291Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-12-20T04:15:12.662578604Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-12-20T04:15:12.662605697Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_TYPE=sqlite3"
logger=settings t=2022-12-20T04:15:12.662632568Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-12-20T04:15:12.662658035Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-12-20T04:15:12.662683229Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-12-20T04:15:12.662718377Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-12-20T04:15:12.66274584Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-12-20T04:15:12.6627715Z level=info msg="App mode production"
logger=sqlstore t=2022-12-20T04:15:12.662938787Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-12-20T04:15:12.662989201Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxr-xr-x expected=-rw-r-----
logger=migrator t=2022-12-20T04:15:12.71598385Z level=info msg="Starting DB migrations"
logger=migrator t=2022-12-20T04:15:12.734386423Z level=info msg="migrations completed" performed=0 skipped=443 duration=1.184758ms

Owner of /var/lib/grafana is grafana:grafana. The telegraf service is running, but reporting errors writing to influxdb.

2022-12-20T04:05:29Z I! Starting Telegraf 1.16.3
2022-12-20T04:05:29Z I! Loaded inputs: cpu disk diskio kernel mem processes snmp statsd swap syslog system
2022-12-20T04:05:29Z I! Loaded aggregators:
2022-12-20T04:05:29Z I! Loaded processors:
2022-12-20T04:05:29Z I! Loaded outputs: influxdb
2022-12-20T04:05:29Z I! Tags enabled: host=telegraf-influxdb-grafana
2022-12-20T04:05:29Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf-influxdb-grafana", Flush Interval:10s
2022-12-20T04:05:29Z I! [inputs.statsd] UDP listening on "[::]:8125"
2022-12-20T04:05:29Z I! [inputs.statsd] Started the statsd service on ":8125"
2022-12-20T04:05:44Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-12-20T04:05:44Z E! [agent] Error writing to outputs.influxdb: could not write any address

Any idea what's happening? This was tested with the same result both on latest and develop.

@pad92
Copy link

pad92 commented Jan 16, 2023

duplicate with #8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants