GESIS server is unstable

[rgaiacs]

@arnim and @rgaiacs noticed that the GESIS server is unstable since July 31, 2024 at 6:29:45 PM GMT+2.

@arnim and @rgaiacs are in contact with GESIS IT to resolve the network issue that is causing the server to be unstable. Unfortunately, part of GESIS IT is on summer holiday and resolve the network issue will take longer.

Related Topics in Jupyter Community Forum

• Github repo loading failure - container already present on machine? from 6 Aug 2024

[rgaiacs]

What @arnim and I know based on the logs (copy at the end of this message) is that the server is taking very long to pull some images.

GESIS server pulled gesiscss/binder-r2d-g5b5b759-sussmanlab-2dopen-2dqmin-395716 (701MB) in less than 2min. This looks very reasonable.

But GESIS server pulled gesiscss/binder-r2d-g5b5b759-serhatcevikel-2dbeyond-5fbounds-cb7907 (2.5GB) in 15min instead of the "expected" 8min.

The very long time to pull some image is creating a queue on Kubernetes.

Kubernetes Event Log

58m                 Normal    Pulled             Pod/jupyter-alliander-2dopens-2doffline-2dexample-2d0wnqtiz3          Successfully pulled image "gesiscss/binder-r2d-g5b5b759-alliander-2dopensource-2dopenstf-2doffline-2dexample-fec7e5:dff747a2312584bfd1f949df1b9bb25a241635b4" in 9m31.407s (31m4.884s including waiting)
53m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2dnq318vi0      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 5m15.152s (34m49.115s including waiting)
53m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2dcon5513w      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 959ms (34m11.684s including waiting)
52m                 Normal    Pulled             Pod/jupyter-mehmetgoekce-2dpandasrepo-2d3yabo9nn                      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-mehmetgoekce-2dpandasrepo-289428:ad60831eb7134f95f69b27ae9d95e5d1ee13fa79" in 4.853s (33m50.22s including waiting)
52m                 Normal    Pulled             Pod/jupyter-seasfire-2dseasfire-2ddatacube-2d6njuj8kx                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-seasfire-2dseasfire-2ddatacube-fc34a3:e4559c252e60ce77c2dac066f09572f6dcd29020" in 6.81s (33m53.99s including waiting)
52m                 Normal    Pulled             Pod/jupyter-seasfire-2dseasfire-2ddatacube-2dag08i75b                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-seasfire-2dseasfire-2ddatacube-fc34a3:e4559c252e60ce77c2dac066f09572f6dcd29020" in 897ms (33m54.884s including waiting)
52m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2d2wvqx292      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 914ms (32m18.822s including waiting)
52m                 Normal    Pulled             Pod/jupyter-seasfire-2dseasfire-2ddatacube-2dzyfs85sr                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-seasfire-2dseasfire-2ddatacube-fc34a3:e4559c252e60ce77c2dac066f09572f6dcd29020" in 947ms (32m19.766s including waiting)
52m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2d7aiwkqk7      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 931ms (32m20.694s including waiting)
38m                 Normal    Pulled             Pod/jupyter-serhatcevikel-2dbeyond-5fbounds-2d4bb1weee                Successfully pulled image "gesiscss/binder-r2d-g5b5b759-serhatcevikel-2dbeyond-5fbounds-cb7907:707c972c92a2b2302e5bf2b1ff37e1897fc05566" in 14m13.038s (46m27.661s including waiting)
38m                 Normal    Pulled             Pod/jupyter-tamaas-2dtutorials-2d85hm0n34                             Successfully pulled image "gesiscss/binder-r2d-g5b5b759-tamaas-2dtutorials-57d971:afab8caddea9ec243328e86728160fce9f23d1d6" in 972ms (46m5.389s including waiting)
35m                 Normal    Pulled             Pod/jupyter-ient-2dgdet3-2ddemos-2d9k7jeny1                           Successfully pulled image "gesiscss/binder-r2d-g5b5b759-https-3a-2f-2fgit-2erwth-2daachen-2ede-2fient-2fgdet3-2ddemos-ba6e39:815809ef1fe2e618ecf179aa83a58c602734e0cd" in 3m7.648s (48m6.352s including waiting)
35m                 Normal    Pulled             Pod/jupyter-seasfire-2dseasfire-2ddatacube-2dmiopy1c1                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-seasfire-2dseasfire-2ddatacube-fc34a3:e4559c252e60ce77c2dac066f09572f6dcd29020" in 967ms (48m5.294s including waiting)
35m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2de3jos8nc      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 904ms (46m39.273s including waiting)
35m                 Normal    Pulled             Pod/jupyter-tamaas-2dtutorials-2dx466m43a                             Successfully pulled image "gesiscss/binder-r2d-g5b5b759-tamaas-2dtutorials-57d971:afab8caddea9ec243328e86728160fce9f23d1d6" in 899ms (46m40.168s including waiting)
35m                 Normal    Pulled             Pod/jupyter-serhatcevikel-2dbeyond-5fbounds-2dgq9gi4kq                Successfully pulled image "gesiscss/binder-r2d-g5b5b759-serhatcevikel-2dbeyond-5fbounds-cb7907:707c972c92a2b2302e5bf2b1ff37e1897fc05566" in 976ms (46m38.102s including waiting)
35m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2dgx8wnz0z      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 952ms (45m40.457s including waiting)
35m                 Normal    Pulled             Pod/jupyter-gesis-2dmethods-2dh-2d-2drstats-2dpackage-2dlkxulh3z      Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gesis-2dmethods-2dhub-2dminimal-2dexample-2drstats-2dpackage-76586d:5071c91770e585defd36eaece066ccf3f3f9a29e" in 909ms (45m0.946s including waiting)
35m                 Normal    Pulled             Pod/jupyter-seasfire-2dseasfire-2ddatacube-2d789zfvgb                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-seasfire-2dseasfire-2ddatacube-fc34a3:e4559c252e60ce77c2dac066f09572f6dcd29020" in 918ms (43m44.02s including waiting)
33m                 Normal    Pulled             Pod/jupyter-sussmanlab-2dopen-2dqmin-2dhthuk772                       Successfully pulled image "gesiscss/binder-r2d-g5b5b759-sussmanlab-2dopen-2dqmin-395716:63006a69bffcae7cb6758c80f6f78bf89fefbe54" in 1m38.282s (45m14.216s including waiting)
24m                 Normal    Pulled             Pod/jupyter-asfadmin-2dasf-2djupyter-2dnotebooks-2dsjysjscm           Successfully pulled image "gesiscss/binder-r2d-g5b5b759-asfadmin-2dasf-2djupyter-2dnotebooks-48f136:9a699e6e1dbdd3ca29dc16446759edcc62df7c61" in 8m46.078s (51m49.817s including waiting)
24m                 Normal    Pulled             Pod/jupyter-asfadmin-2dasf-2djupyter-2dnotebooks-2drkb7pf1z           Successfully pulled image "gesiscss/binder-r2d-g5b5b759-asfadmin-2dasf-2djupyter-2dnotebooks-48f136:9a699e6e1dbdd3ca29dc16446759edcc62df7c61" in 1.015s (51m8.366s including waiting)
24m                 Normal    Pulled             Pod/jupyter-asfadmin-2dasf-2djupyter-2dnotebooks-2de7aav8sg           Successfully pulled image "gesiscss/binder-r2d-g5b5b759-asfadmin-2dasf-2djupyter-2dnotebooks-48f136:9a699e6e1dbdd3ca29dc16446759edcc62df7c61" in 904ms (50m52.058s including waiting)
24m                 Normal    Pulled             Pod/jupyter-svonworl-2dsimple-2dnotebook-2dckux3s2v                   Successfully pulled image "gesiscss/binder-r2d-g5b5b759-svonworl-2dsimple-2dnotebook-465f11:daefecc62f2aebc18c216b474508a1e61edf832f" in 3.218s (50m27.979s including waiting)
24m                 Normal    Pulled             Pod/jupyter-seasfire-2dseasfire-2ddatacube-2d8c70hflj                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-seasfire-2dseasfire-2ddatacube-fc34a3:e4559c252e60ce77c2dac066f09572f6dcd29020" in 1.005s (49m17.203s including waiting)
17m                 Normal    Pulled             Pod/jupyter-vgvassilev-2dclad-2dl3dv9juc                              Successfully pulled image "gesiscss/binder-r2d-g5b5b759-vgvassilev-2dclad-0d1fc4:1b81084bd69b15c776582e24aa5d7bbe411c5b81" in 7m35.802s (56m33.797s including waiting)

[rgaiacs]

GESIS server continues to be very unstable.

Grafana reports the following during the first 12 hours of last Monday (19 August 2024):

The high number of Pending pods (yellow line) has been a constant since this issue was open.

Grafana reports the following during the first 12 hours of this Monday (26 August 2024):

Something happen that the number of Pending Pods (yellow line) was small on Monday morning.

The point of inflection was on Saturday, 24 August 2024 around 11:00AM. My hypothesis is that something changed in the GESIS network but I'm waiting for confirmation.

Unfortunately, around 8:15 of today (27 August 2024), the Pending pods started to be very high.

[rgaiacs]

The number of pending pods grow very fast on even after stop the server for a while.

I looked at the event log and the waiting time to pull images keeps increasing:

0s                    Normal    Pulled              Pod/jupyter-npa-2djnotebooks-2dlab-5fgeo-5felec01-2dmb8zt99h       Successfully pulled image "gesiscss/binder-r2d-g5b5b759-npa-5f-2djnotebooks-2dlab-5fgeo-5felec01-6c9b4e:07d9c3b94637a4c29fa41e356055b7b6cdd52042" in 10.438s (10.438s including waiting)
0s                    Normal    Pulled              Pod/jupyter-noamross-2dgams-2din-2dr-2dcourse-2duowijrbg              Successfully pulled image "gesiscss/binder-r2d-g5b5b759-noamross-2dgams-2din-2dr-2dcourse-1fc161:ac4c2572f553e1365e2d3eb16fe8db548fe689f1" in 3m22.422s (3m22.422s including waiting)
0s                    Normal    Pulled              Pod/jupyter-rhodyprog4ds-2dbrownfall21-2d59t8fqk4                     Successfully pulled image "gesiscss/binder-r2d-g5b5b759-rhodyprog4ds-2dbrownfall21-a2343d:433500586ca5565bef504ff4c27f320bf46664a3" in 1m28.63s (4m39.973s including waiting)
0s                    Normal    Pulled              Pod/jupyter-yuanluya-2dcl-5fhuman-2d183hqfmd                          Successfully pulled image "gesiscss/binder-r2d-g5b5b759-yuanluya-2dcl-5fhuman-eebc76:c9f4fe3981ba93353c9180485cfb28597392f919" in 2m6.36s (6m29.216s including waiting)
0s                    Normal    Pulled              Pod/jupyter-esterodr-2dtp-5festadistica-2dek84lnlp                    Successfully pulled image "gesiscss/binder-r2d-g5b5b759-esterodr-2dtp-5festadistica-aed387:a1e119cb877432c6be42b251090a8418fc98b8d2" in 2m31.837s (8m29.788s including waiting)
0s                    Normal    Pulled              Pod/jupyter-maxrudolph-2dg326-2dstudent-2dbkc44zq7                    Successfully pulled image "gesiscss/binder-r2d-g5b5b759-maxrudolph-2dg326-2dstudent-e06f01:e39151caa8c073b1f3ce689529c0898118f3a37c" in 54.307s (9m5.934s including waiting)
0s                    Normal    Pulled              Pod/jupyter-yuanluya-2dcl-5fhuman-2dz5z03qsq                          Successfully pulled image "gesiscss/binder-r2d-g5b5b759-yuanluya-2dcl-5fhuman-eebc76:c9f4fe3981ba93353c9180485cfb28597392f919" in 924ms (8m37.64s including waiting)
0s                    Normal    Pulled              Pod/jupyter-yuanluya-2dcl-5fhuman-2dnd660tlb                          Successfully pulled image "gesiscss/binder-r2d-g5b5b759-yuanluya-2dcl-5fhuman-eebc76:c9f4fe3981ba93353c9180485cfb28597392f919" in 920ms (8m26.454s including waiting)
0s                    Normal    Pulled              Pod/jupyter-esterodr-2dtp-5festadistica-2d78xbpli8                    Successfully pulled image "gesiscss/binder-r2d-g5b5b759-esterodr-2dtp-5festadistica-aed387:a1e119cb877432c6be42b251090a8418fc98b8d2" in 907ms (8m13.237s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-deathbeds-2dhourhaus-2dvxmranfb                           Successfully pulled image "gesiscss/binder-r2d-g5b5b759-deathbeds-2dhourhaus-8a15f3:66cc05e92fe86800393a939366ce82c1fdd181de" in 4m29.155s (12m3.078s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-opendata-2deducat-2dhiukkasfysiikka-2dd7y45qgf            Successfully pulled image "gesiscss/binder-r2d-g5b5b759-opendata-2deducation-2dhiukkasfysiikka-3d0321:2e397a2baeca544b23211d67cfb2093235871209" in 1m38.175s (12m54.855s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-deathbeds-2dhourhaus-2dtz0ue8it                           Successfully pulled image "gesiscss/binder-r2d-g5b5b759-deathbeds-2dhourhaus-8a15f3:66cc05e92fe86800393a939366ce82c1fdd181de" in 987ms (12m31.586s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-aparrish-2drwet-2d2f52uyyq                                Successfully pulled image "gesiscss/binder-r2d-g5b5b759-aparrish-2drwet-4057e2:6dd26af2205e9c81b6c7589bbe20784773a0e757" in 9m55.942s (22m8.338s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-aparrish-2drwet-2dvy4edgzd                                Successfully pulled image "gesiscss/binder-r2d-g5b5b759-aparrish-2drwet-4057e2:6dd26af2205e9c81b6c7589bbe20784773a0e757" in 1.006s (20m50.553s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-globus-2dglobus-2dsdk-2dpython-2d72jdw3f2                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-globus-2dglobus-2dsdk-2dpython-e6b934:ecfc638b497cf8613aeade149a917ef9e3bf7d43" in 4.741s (19m18.361s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-globus-2dglobus-2dsdk-2dpython-2dt0066adg                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-globus-2dglobus-2dsdk-2dpython-e6b934:ecfc638b497cf8613aeade149a917ef9e3bf7d43" in 897ms (19m18.243s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-aimacode-2daima-2dpython-2dixiogtc5                       Successfully pulled image "gesiscss/binder-r2d-g5b5b759-aimacode-2daima-2dpython-249bb8:61d695b37c6895902081da1f37baf645b0d2658a" in 3m17.268s (21m35.932s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-scikit-2dlearn-2dscikit-2dlearn-2do4srnl6i                Successfully pulled image "gesiscss/binder-r2d-g5b5b759-scikit-2dlearn-2dscikit-2dlearn-4c9324:3f89022fa04d293152f1d32fbc2a5bdaaf2df364" in 2m20.634s (22m26.73s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-mmoelle1-2doospcpp-2d8pr5mfq6                             Successfully pulled image "gesiscss/binder-r2d-g5b5b759-mmoelle1-2doospcpp-22a83d:b247f3272ce728186bb1ea5f7441ee3f61604ed2" in 3m32.813s (25m36.331s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-globus-2dglobus-2dsdk-2dpython-2djfwahzat                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-globus-2dglobus-2dsdk-2dpython-e6b934:ecfc638b497cf8613aeade149a917ef9e3bf7d43" in 954ms (24m50.794s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-gis-2dops-2drouting-2dpy-2dyzilc25t                       Successfully pulled image "gesiscss/binder-r2d-g5b5b759-gis-2dops-2drouting-2dpy-560beb:a86cee6abad26fd9d95a131a68c2dc00ef738131" in 1m47.093s (26m10.575s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-mmoelle1-2doospcpp-2dqgpoa6yo                             Successfully pulled image "gesiscss/binder-r2d-g5b5b759-mmoelle1-2doospcpp-22a83d:b247f3272ce728186bb1ea5f7441ee3f61604ed2" in 944ms (25m41.276s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-globus-2dglobus-2dsdk-2dpython-2d5l0x175p                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-globus-2dglobus-2dsdk-2dpython-e6b934:ecfc638b497cf8613aeade149a917ef9e3bf7d43" in 939ms (25m33.147s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-ruby-2dnumo-2dnumo-2dlinalg-2d5lqk4631                    Successfully pulled image "gesiscss/binder-r2d-g5b5b759-ruby-2dnumo-2dnumo-2dlinalg-d1addf:df12cfbcddf42f66bb3acd93bec93c554093d6a1" in 3.898s (25m20.889s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-globus-2dglobus-2dsdk-2dpython-2depuq5a4z                 Successfully pulled image "gesiscss/binder-r2d-g5b5b759-globus-2dglobus-2dsdk-2dpython-e6b934:ecfc638b497cf8613aeade149a917ef9e3bf7d43" in 945ms (24m30.313s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-coderbee-2dsculpt-2d80end1se                              Successfully pulled image "gesiscss/binder-r2d-g5b5b759-coderbee-2dsculpt-dcd920:d12c24dea3e8fa56b45054d7040773dfeb634255" in 3.61s (21m52.725s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-isaac-2dflath-2dfastblog-2d2yt7yr4a                       Successfully pulled image "gesiscss/binder-r2d-g5b5b759-isaac-2dflath-2dfastblog-f624ed:4f477836507e40a18d29c156ebc535cb9d89b6fe" in 1m23.746s (21m44.745s including waiting)
0s                    Normal    Pulled                   Pod/jupyter-coffeateam-2dcoffea-2dcy7p41ts                            Successfully pulled image "gesiscss/binder-r2d-g5b5b759-coffeateam-2dcoffea-ec6eae:4de015975bd261c294c28efc1a0c51a46d7d11e2" in 1m8.244s (22m3.612s including waiting)

[rgaiacs]

The GESIS server got to a point that the number of pending / terminating pods are locking the Kubernetes Cluster. 😭

@arnim and I will have to reduce the number of users that we can serve.