[lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get fromams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer

[patsevanton]

Thanks for docker-compose-healthcheck

docker-compose up with PR #17 and #19

Get error

kong            | 2022/02/15 04:31:42 [crit] 1104#0: *14 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer

Full log

docker-compose up
[+] Running 4/4
 - Network docker-compose-healthcheck_default  Created                                                                                0.1s 
 - Container kong-postgres                     Created                                                                                0.1s 
 - Container kong-migration                    Created                                                                                0.1s
 - Container kong                              Created                                                                                0.1s
Attaching to kong, kong-migration, kong-postgres
kong-postgres   | ********************************************************************************
kong-postgres   | WARNING: POSTGRES_HOST_AUTH_METHOD has been set to "trust". This will allow
kong-postgres   |          anyone with access to the Postgres port to access your database without
kong-postgres   |          a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
kong-postgres   |          documentation about "trust":
kong-postgres   |          https://www.postgresql.org/docs/current/auth-trust.html
kong-postgres   |          In Docker's default configuration, this is effectively any other
kong-postgres   |          container on the same system.
kong-postgres   |
kong-postgres   |          It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
kong-postgres   |          it with "-e POSTGRES_PASSWORD=password" instead to set a password in
kong-postgres   |          "docker run".
kong-postgres   | ********************************************************************************
kong-postgres   | The files belonging to this database system will be owned by user "postgres".
kong-postgres   | This user must also own the server process.
kong-postgres   |
kong-postgres   | The database cluster will be initialized with locale "en_US.utf8".
kong-postgres   | The default database encoding has accordingly been set to "UTF8".
kong-postgres   | The default text search configuration will be set to "english".
kong-postgres   |
kong-postgres   | Data page checksums are disabled.
kong-postgres   |
kong-postgres   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
kong-postgres   | creating subdirectories ... ok
kong-postgres   | selecting default max_connections ... 100
kong-postgres   | selecting default shared_buffers ... 128MB
kong-postgres   | selecting default timezone ... Etc/UTC
kong-postgres   | selecting dynamic shared memory implementation ... posix
kong-postgres   | creating configuration files ... ok
kong-postgres   | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
kong-postgres   | initializing pg_authid ... ok
kong-postgres   | setting password ... ok
kong-postgres   | initializing dependencies ... ok
kong-postgres   | creating system views ... ok
kong-postgres   | loading system objects' descriptions ... ok
kong-postgres   | creating collations ... ok
kong-postgres   | creating conversions ... ok
kong-postgres   | creating dictionaries ... ok
kong-postgres   | setting privileges on built-in objects ... ok
kong-postgres   | creating information schema ... ok
kong-postgres   | loading PL/pgSQL server-side language ... ok
kong-postgres   | vacuuming database template1 ... ok
kong-postgres   | copying template1 to template0 ... ok
kong-postgres   | copying template1 to postgres ... ok
kong-postgres   | syncing data to disk ... ok
kong-postgres   |
kong-postgres   | Success. You can now start the database server using:
kong-postgres   |
kong-postgres   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
kong-postgres   |
kong-postgres   |
kong-postgres   | WARNING: enabling "trust" authentication for local connections
kong-postgres   | You can change this by editing pg_hba.conf or using the option -A, or
kong-postgres   | --auth-local and --auth-host, the next time you run initdb.
kong-postgres   | waiting for server to start....LOG:  database system was shut down at 2022-02-15 04:31:23 UTC
kong-postgres   | LOG:  MultiXact member wraparound protections are now enabled
kong-postgres   | LOG:  autovacuum launcher started
kong-postgres   | LOG:  database system is ready to accept connections
kong-postgres   |  done
kong-postgres   | server started
kong-postgres   | CREATE DATABASE
kong-postgres   | 
kong-postgres   |
kong-postgres   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
kong-postgres   |
kong-postgres   | waiting for server to shut down...LOG:  received fast shutdown request
kong-postgres   | .LOG:  aborting any active transactions
kong-postgres   | LOG:  autovacuum launcher shutting down
kong-postgres   | LOG:  shutting down
kong-postgres   | LOG:  database system is shut down
kong-postgres   |  done
kong-postgres   | server stopped
kong-postgres   |
kong-postgres   | PostgreSQL init process complete; ready for start up.
kong-postgres   |
kong-postgres   | LOG:  database system was shut down at 2022-02-15 04:31:25 UTC
kong-postgres   | LOG:  MultiXact member wraparound protections are now enabled
kong-postgres   | LOG:  autovacuum launcher started
kong-postgres   | LOG:  database system is ready to accept connections
kong-migration  | Bootstrapping database...
kong-migration  | migrating core on database 'kong'...
kong-migration  | core migrated up to: 000_base (executed)
kong-migration  | core migrated up to: 003_100_to_110 (executed)
kong-migration  | core migrated up to: 004_110_to_120 (executed)
kong-migration  | core migrated up to: 005_120_to_130 (executed)
kong-migration  | core migrated up to: 006_130_to_140 (executed)
kong-migration  | core migrated up to: 007_140_to_150 (executed)
kong-migration  | core migrated up to: 008_150_to_200 (executed)
kong-migration  | core migrated up to: 009_200_to_210 (executed)
kong-migration  | core migrated up to: 010_210_to_211 (executed)
kong-migration  | core migrated up to: 011_212_to_213 (executed)
kong-migration  | core migrated up to: 012_213_to_220 (executed)
kong-migration  | core migrated up to: 013_220_to_230 (executed)
kong-migration  | core migrated up to: 014_230_to_270 (executed)
kong-migration  | migrating acl on database 'kong'...
kong-migration  | acl migrated up to: 000_base_acl (executed)
kong-migration  | acl migrated up to: 002_130_to_140 (executed)
kong-migration  | acl migrated up to: 003_200_to_210 (executed)
kong-migration  | acl migrated up to: 004_212_to_213 (executed)
kong-migration  | migrating acme on database 'kong'...
kong-migration  | acme migrated up to: 000_base_acme (executed)
kong-migration  | migrating basic-auth on database 'kong'...
kong-migration  | basic-auth migrated up to: 000_base_basic_auth (executed)
kong-migration  | basic-auth migrated up to: 002_130_to_140 (executed)
kong-migration  | basic-auth migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating bot-detection on database 'kong'...
kong-migration  | bot-detection migrated up to: 001_200_to_210 (executed)
kong-migration  | migrating hmac-auth on database 'kong'...
kong-migration  | hmac-auth migrated up to: 000_base_hmac_auth (executed)
kong-migration  | hmac-auth migrated up to: 002_130_to_140 (executed)
kong-migration  | hmac-auth migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating ip-restriction on database 'kong'...
kong-migration  | ip-restriction migrated up to: 001_200_to_210 (executed)
kong-migration  | migrating jwt on database 'kong'...
kong-migration  | jwt migrated up to: 000_base_jwt (executed)
kong-migration  | jwt migrated up to: 002_130_to_140 (executed)
kong-migration  | jwt migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating key-auth on database 'kong'...
kong-migration  | key-auth migrated up to: 000_base_key_auth (executed)
kong-migration  | key-auth migrated up to: 002_130_to_140 (executed)
kong-migration  | key-auth migrated up to: 003_200_to_210 (executed)
kong-migration  | migrating oauth2 on database 'kong'...
kong-migration  | oauth2 migrated up to: 000_base_oauth2 (executed)
kong-migration  | oauth2 migrated up to: 003_130_to_140 (executed)
kong-migration  | oauth2 migrated up to: 004_200_to_210 (executed)
kong-migration  | oauth2 migrated up to: 005_210_to_211 (executed)
kong-migration  | migrating rate-limiting on database 'kong'...
kong-migration  | rate-limiting migrated up to: 000_base_rate_limiting (executed)
kong-migration  | rate-limiting migrated up to: 003_10_to_112 (executed)
kong-migration  | rate-limiting migrated up to: 004_200_to_210 (executed)
kong-migration  | migrating response-ratelimiting on database 'kong'...
kong-migration  | response-ratelimiting migrated up to: 000_base_response_rate_limiting (executed)
kong-migration  | migrating session on database 'kong'...
kong-migration  | session migrated up to: 000_base_session (executed)
kong-migration  | session migrated up to: 001_add_ttl_index (executed)
kong-migration  | 42 migrations processed
kong-migration  | 42 executed
kong-migration  | Database is up-to-date
kong-migration exited with code 0
kong            | 2022/02/15 04:31:37 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
kong            | nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
kong            | 2022/02/15 04:31:37 [notice] 1#0: using the "epoll" event method
kong            | 2022/02/15 04:31:37 [notice] 1#0: openresty/1.19.9.1
kong            | 2022/02/15 04:31:37 [notice] 1#0: built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
kong            | 2022/02/15 04:31:37 [notice] 1#0: OS: Linux 5.10.60.1-microsoft-standard-WSL2
kong            | 2022/02/15 04:31:37 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker processes
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1098
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1099
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1100
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1101
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1102
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1103
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1104
kong            | 2022/02/15 04:31:37 [notice] 1#0: start worker process 1105
kong            | 2022/02/15 04:31:37 [notice] 1100#0: *3 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1098#0: *1 [lua] warmup.lua:92: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1103#0: *6 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1099#0: *2 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1105#0: *8 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1104#0: *7 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1098#0: *1 [lua] warmup.lua:129: single_dao(): finished preloading 'services' into the core_cache (in 0ms), context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1101#0: *4 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:37 [notice] 1102#0: *5 [kong] init.lua:311 only worker #0 can manage, context: init_worker_by_lua*
kong            | 2022/02/15 04:31:42 [crit] 1104#0: *14 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:31:42 [crit] 1099#0: *10 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:31:42 [crit] 1105#0: *15 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:31:42 [crit] 1100#0: *11 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:31:42 [crit] 1098#0: *16 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:31:42 [crit] 1101#0: *17 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer
kong            | 2022/02/15 04:31:42 [crit] 1102#0: *18 [lua] balancers.lua:240: create_balancers(): failed loading initial list of upstreams: failed to get from node cache: could not acquire callback lock: timeout, context: ngx.timer

[patsevanton]

Same issue

Kong/kong#3613

Kong/kong#5837

https://discuss.konghq.com/t/kong-deploy-in-kubernetes-db-mode-getting-dns-resolution-failed/5796/12

https://stackoverflow.com/questions/69501203/kong-with-docker-compose-on-windows-failed-loading-initial-list-of-upstreams-f