Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RichDocuments built-in CODE causes significant slowdown/timeout issues #1282

Open
eganonoa opened this issue Dec 7, 2020 · 52 comments · May be fixed by #3762
Open

RichDocuments built-in CODE causes significant slowdown/timeout issues #1282

eganonoa opened this issue Dec 7, 2020 · 52 comments · May be fixed by #3762
Assignees
Labels
2. developing Work in progress bug Something isn't working high integration: built-in code server richdocumentscode performance

Comments

@eganonoa
Copy link

eganonoa commented Dec 7, 2020

Over the last couple of weeks Snap and Docker versions of Nextcloud have updated to version 2.0.2 (snap on Friday). After the upgrade the connection to Nextcloud, particularly via the web-interface, slows to a crawl and/or times out. It appears to be caused by richdocuments/collabora.

What is strange about this is that you cannot see easily identify the problem: basic analytics show no major increase in load on the CPU, Ram or I/O, the logs don't immediately show anything out of the ordinary. If you open top/htop on the server, however, and observe for a while you can see big, temporary spikes where available CPU's are maxed out by richdocuments/the Appimage related to Collabora and the Code server. When you go to the admin settings under collabora you can see an error that says that it cannot establish a connection to the in-built CODE server.

The performance problem can be fixed by disabling both the collabora and code server apps: either in the web interface if you can get there through the performance issues, or in the server terminal (sudo snap run nextcloud.occ app:disable richdocuments ; sudo snap run nextcloud.occ app:disable richdocumentscore). After doing that normal service is resumed entirely across the Nextcloud server.

This indicates that the failure to establish a connection to the inbuilt CODE server is the underlying problem and that there is something that happens when upgrading from Nextcloud v. 2.0.1 to 2.0.2 that causes it to break. Strangely, however, this is an issue that only appears to be present in Nextcloud Snap (where slowdown occurs) and Docker (where timeouts occur) and doesn't appear to have been reported by users of the regular Nextcloud: see threads on Nextcloud Community Forum (https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304), Nextcloud Snap Github (nextcloud-snap/nextcloud-snap#1569), and Nextcloud Reddit (https://www.reddit.com/r/NextCloud/comments/k7b1of/extremely_slow_interface_nextcloud_snap/).

@calbav
Copy link

calbav commented Dec 7, 2020

Hello,

my Nextcloud instance is experiencing the same issue after version upgrade, although in my case it is enough to disable the Collabora Online - Built-in CODE Server app to solve the performance problem.

Environment:
Ubuntu 18.04.5 LTS
snap 20.0.2snap2

Thank you for your time and effort in looking into this matter. Should you need additional information please let me know.

@kyrofa
Copy link
Member

kyrofa commented Dec 7, 2020

Confirmed, this makes the snap unusable. Nothing interesting in the logs during usage that doesn't relate to collabora. When trying to open a document I see this though. Not sure if it's helpful:

{"reqId":"FX5iHBU58fJlpkNjGQsY","level":3,"time":"2020-12-07T17:38:50+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/index?fileId=11&requesttoken=AEyf86%2BIbFplFbZnaNJpmAjUL4WrtY98go6EYe6v%2BNM%3D%3ANDjKqtXyKw8dW9QFX7gBt2nlH%2FTYhd447v7HL6jXteE%3D","message":{"Exception":"Exception","Message":"Could not find urlsrc in WOPI","Code":0,"Trace":[{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/TokenManager.php","line":210,"function":"getUrlSrc","class":"OCA\\Richdocuments\\WOPI\\Parser","type":"->","args":["application/vnd.openxmlformats-officedocument.wordprocessingml.document"]},{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/Controller/DocumentController.php","line":256,"function":"getToken","class":"OCA\\Richdocuments\\TokenManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":169,"function":"index","class":"OCA\\Richdocuments\\Controller\\DocumentController","type":"->","args":["*** sensitive parameter replaced ***",null]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/App.php","line":152,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/Route/Router.php","line":308,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Richdocuments\\Controller\\DocumentController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"action":null,"_route":"richdocuments.document.index"}]},{"file":"/snap/nextcloud/24828/htdocs/lib/base.php","line":1008,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/richdocuments/index"]},{"file":"/snap/nextcloud/24828/htdocs/index.php","line":37,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/WOPI/Parser.php","Line":54,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0","version":"20.0.2.2"}

Note that I've been trying to open that document for about 5 minutes now, it just keeps spinning.

@kyrofa
Copy link
Member

kyrofa commented Dec 7, 2020

Got a lot of these in my PHP error log:

[07-Dec-2020 17:38:41 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208
[07-Dec-2020 17:38:43 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208
[07-Dec-2020 17:38:45 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208
[07-Dec-2020 17:38:47 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: Connection timed out in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208

And this type of thing in PHP-FPM error log:

[07-Dec-2020 09:40:44] WARNING: [pool www] child 17933 said into stderr: "File exists and file size matches, skipping"
[07-Dec-2020 09:40:44] WARNING: [pool www] child 17933 said into stderr: "Logging at warning level to file: /tmp/loolwsd.wQQ51O50TX/loolwsd.log"
[07-Dec-2020 09:40:44] WARNING: [pool www] child 15589 said into stderr: "Logging at warning level to file: /tmp/loolwsd.4qc5ViOkqZ/loolwsd.log"
[07-Dec-2020 09:40:44] WARNING: [pool www] child 17933 said into stderr: "Failed to listen on Server port(s) (9982-9982). Exiting."
[07-Dec-2020 09:40:45] WARNING: [pool www] child 15589 said into stderr: "Failed to listen on Server port(s) (9982-9982). Exiting."
[07-Dec-2020 09:40:48] WARNING: [pool www] child 17933 said into stderr: "/tmp/appimage_extracted_9bc52a5da15b4853d238ef00019d6a4d/opt/collaboraoffice6.4/share/autotext/es/standard.bau: ftw error: No such file or directory"
[07-Dec-2020 09:40:48] WARNING: [pool www] child 17933 said into stderr: "Failed to clean up cache directory"

@juliushaertl
Copy link
Member

cc @timar @eszkadev Seems caused or related to richdocumentscode. Any idea?

@cedartop
Copy link

cedartop commented Dec 7, 2020

I found something on the nextcloud forum:

hpilkington gave the tip:

Confirmed sudo snap run nextcloud.occ app:disable richdocuments && sudo snap run nextcloud.occ app:disable richdocumentscore resolved my issues.

After having done that, the web interface turns fast again. I hope there will be a fix soon for collabora...

@eszkadev
Copy link
Contributor

eszkadev commented Dec 8, 2020

Confirmed, this makes the snap unusable. Nothing interesting in the logs during usage that doesn't relate to collabora. When trying to open a document I see this though. Not sure if it's helpful:

{"reqId":"FX5iHBU58fJlpkNjGQsY","level":3,"time":"2020-12-07T17:38:50+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/index?fileId=11&requesttoken=AEyf86%2BIbFplFbZnaNJpmAjUL4WrtY98go6EYe6v%2BNM%3D%3ANDjKqtXyKw8dW9QFX7gBt2nlH%2FTYhd447v7HL6jXteE%3D","message":{"Exception":"Exception","Message":"Could not find urlsrc in WOPI","Code":0,"Trace":[{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/TokenManager.php","line":210,"function":"getUrlSrc","class":"OCA\\Richdocuments\\WOPI\\Parser","type":"->","args":["application/vnd.openxmlformats-officedocument.wordprocessingml.document"]},{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/Controller/DocumentController.php","line":256,"function":"getToken","class":"OCA\\Richdocuments\\TokenManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":169,"function":"index","class":"OCA\\Richdocuments\\Controller\\DocumentController","type":"->","args":["*** sensitive parameter replaced ***",null]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/App.php","line":152,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/Route/Router.php","line":308,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Richdocuments\\Controller\\DocumentController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"action":null,"_route":"richdocuments.document.index"}]},{"file":"/snap/nextcloud/24828/htdocs/lib/base.php","line":1008,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/richdocuments/index"]},{"file":"/snap/nextcloud/24828/htdocs/index.php","line":37,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/WOPI/Parser.php","Line":54,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0","version":"20.0.2.2"}

Note that I've been trying to open that document for about 5 minutes now, it just keeps spinning.

@juliushaertl I noticed similar issue probably introduced by: 6b3d3f1
Request for "/index.php/apps/richdocuments/index" hangs in DocumentController.php because we need discovery while getting token and after mentioned patch there is no local cache which may be used so we are trying to connect to richdocumentscode which may be not started yet. But it should be fixed now in my last commits by increasing timeout when it is starting.

@thebearon
Copy link
Collaborator

thebearon commented Dec 8, 2020

Looked into the snap case with @timar. The findings so far (spoiler: the interesting part is at the end):
/var/snap/nextcloud/24828/log/php-fpm_errors.log has relevant content.

[08-Dec-2020 19:56:50] NOTICE: fpm is running, pid 3042
[08-Dec-2020 19:56:50] NOTICE: ready to handle connections

So far so good.

[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "sh: 1: pidof: Permission denied"
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "sh: 1: disown: not found"

These aren't ideal, and might be worth investigating some other time (service may fail to kill the old version, and start the new one after an update), but isn't relevant now.

[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "dlopen(): error loading libfuse.so.2"
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "AppImages require FUSE to run. "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "You might still be able to extract the contents of this AppImage "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "if you run it with the --appimage-extract option. "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "See https://github.com/AppImage/AppImageKit/wiki/FUSE "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "for more information"

This is expected, there's no libfuse2 in snap, content of the AppImage is extracted to /tmp/snap.nextcloud/tmp/.

[08-Dec-2020 22:08:51] WARNING: [pool www] child 20232 said into stderr: "Logging at warning level to file: /tmp/loolwsd.2y1F0DbTKo/loolwsd.log"

loolwsd has started.

[08-Dec-2020 22:08:51] WARNING: [pool www] child 20232 said into stderr: "Error: loolforkit incorrect user-name: root - aborting"
[08-Dec-2020 22:08:51] WARNING: [pool www] child 20232 said into stderr: "Security: --disable-lool-user-checking failed, loolforkit has some capabilities set."

^ This indicates the problem.
Another indicator: http://<snap host>/extra-apps/richdocumentscode/proxy.php?status stays at {"status":"starting"} instead of OK.

Only happens with version 6.4.202, while the previous 6.4.14 version is okay.
CollaboraOnline/online@2640086 altered a check so !hasCorrectUID("loolforkit") is always called, and emits the first error message, and added another check that causes the app to exit with error after the second error message. I don't know if that's all there is to this problem, but looks like a good starting point.
@kendy, @Ashod, what are your thoughts on that?

@kendy
Copy link
Collaborator

kendy commented Dec 9, 2020

@thebearon: I'd suggest removing the capabilities from the binaries when packaging this, that should make it working again.

@kendy
Copy link
Collaborator

kendy commented Dec 9, 2020

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

@timar
Copy link
Member

timar commented Dec 9, 2020

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

Actually, probably @kyrofa can answer this question.

@juliushaertl
Copy link
Member

The loolwsd command should be ran as the same user that the webserver is using since richdocuments code is just calling PHP exec(), so I assume that this is rather snap specific.

@kyrofa
Copy link
Member

kyrofa commented Dec 9, 2020

Is it really necessary to run us as root?

Yes, as we've discussed before, daemons in snaps run as root, although note that root in snaps is still confined so there are no special capabilities granted.

@eganonoa
Copy link
Author

eganonoa commented Dec 9, 2020

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

Important to note that this issue isn't limited to snap, but has been reported by Docker image users also (though perhaps only those that installed everything as root?). Also note that one user on the Community Forum has said that upgrading to the edge release fixes the issue: https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304/27. It's just one user, of course.

@LouisVallat
Copy link

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

Important to note that this issue isn't limited to snap, but has been reported by Docker image users also (though perhaps only those that installed everything as root?). Also note that one user on the Community Forum has said that upgrading to the edge release fixes the issue: https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304/27. It's just one user, of course.

I use Nextcloud, latest version & fresh install, in Docker with the latest image, which corresponds to the latest stable release. I can't use richdocuments & richdocumentscode because of how slow and how sluggish my instance become when these apps are installed.

@kyrofa kyrofa changed the title RichDocuments causes significant slowdown/timeout issues in Nextcloud 2.0.2 (snap and docker) RichDocuments causes significant slowdown/timeout issues in Nextcloud 20.0.2 Dec 14, 2020
@SimplyCorbett
Copy link

This also happens if nextcloud is installed on bare metal. It's not dependent on docker or snap.

I have a collabora container on another VM, but nextcloud hasn't connected to it yet.

Unfortunately in my case with nextcloud 20.04 the entire install errors out with richdocuments enabled so I can't configure it. With 20.02 I could at least load into the app after 45-60 seconds and once configured the issue went away.

That's not possible on 20.04.

@aaylnx
Copy link

aaylnx commented Dec 22, 2020

I really hope a fix for this is forthcoming soon. I definitely is a big showstopper for me.

@timar
Copy link
Member

timar commented Dec 22, 2020

richdocumentcode 6.4.203 has been released. It fixes the issue, i.e. now we let loolforkit run as root.

@eganonoa
Copy link
Author

richdocumentcode 6.4.203 has been released. It fixes the issue, i.e. now we let loolforkit run as root.

I'm afraid I cannot confirm the fix. I upgraded to 6.4.203 and re-enabled the CODE server and Collabora Online and it has, unfortunately, knocked out my Nextcloud server (latest snap version) again. Disabling them brings performance right back.

@szjozsef
Copy link

szjozsef commented Dec 23, 2020

I can confirm, it still not working, logs:

xx.xx.xx.xx - - [23/Dec/2020:07:37:57 +0000] "GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1" 200 628 "-" "Nextcloud Server Crawler"
dlopen(): error loading libfuse.so.2

AppImages require FUSE to run.
You might still be able to extract the contents of this AppImage
if you run it with the --appimage-extract option.
See https://github.com/AppImage/AppImageKit/wiki/FUSE
for more information

Logging at warning level to file: /tmp/loolwsd.k6vtOuTaTH/loolwsd.log
Error: loolforkit incorrect user-name, other than 'lool' - aborting
Security: --disable-lool-user-checking failed, loolforkit has some capabilities set.

nextcloud 20.0.4 (docker)
Collabora Online 3.7.11
Collabora Online - Built-in CODE Server 6.4.203

@calbav
Copy link

calbav commented Dec 23, 2020

The fix seems to work on snap.

Although everything appears to be working a new notification appears (maybe unrelated?):

"Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files."

Also many new errors are logged, mainly:

[encryption] Warning: OC\ServerNotAvailableException: Legacy cipher is no longer supported! at <>

  1. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Crypto/Crypt.php line 405
    OCA\Encryption\Crypto\Crypt->getLegacyCipher()
  2. /snap/nextcloud/24940/htdocs/apps/encryption/lib/KeyManager.php line 404
    OCA\Encryption\Crypto\Crypt->decryptPrivateKey("*** sensitive parameters replaced ***")
  3. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Hooks/UserHooks.php line 183
    OCA\Encryption\KeyManager->init("*** sensitive parameter replaced ", " sensitive parameter replaced ***")
  4. /snap/nextcloud/24940/htdocs/lib/private/legacy/OC_Hook.php line 110
    OCA\Encryption\Hooks\UserHooks->login("*** sensitive parameters replaced ***")
  5. /snap/nextcloud/24940/htdocs/lib/private/Server.php line 581
    OC_Hook::emit("OC_User", "post_login", "*** sensitive parameter replaced ***")
  6. <>
    OC\Server->OC{closure}("*** sensitive parameters replaced ***")
  7. /snap/nextcloud/24940/htdocs/lib/private/Hooks/EmitterTrait.php line 101
    call_user_func_array(Closure {}, ["*** sensitive ... "])
  8. /snap/nextcloud/24940/htdocs/lib/private/Hooks/PublicEmitter.php line 40
    OC\Hooks\BasicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  9. /snap/nextcloud/24940/htdocs/lib/private/User/Session.php line 903
    OC\Hooks\PublicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  10. /snap/nextcloud/24940/htdocs/lib/base.php line 1058
    OC\User\Session->loginWithCookie("*** sensitive parameters replaced ***")
  11. /snap/nextcloud/24940/htdocs/lib/base.php line 997
    OC::handleLogin(OC\AppFramework\Http\Request {})
  12. /snap/nextcloud/24940/htdocs/index.php line 37
    OC::handleRequest()

@LouisVallat
Copy link

I tested this update this morning, using Docker, and it still acts the same : installing richdocuments & richdocumentscode result is veeerry slow instance, and it goes back to normal after uninstalling these apps.

@eganonoa
Copy link
Author

The fix seems to work on snap.

Although everything appears to be working a new notification appears (maybe unrelated?):

"Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files."

Also many new errors are logged, mainly:

[encryption] Warning: OC\ServerNotAvailableException: Legacy cipher is no longer supported! at <>

  1. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Crypto/Crypt.php line 405
    OCA\Encryption\Crypto\Crypt->getLegacyCipher()
  2. /snap/nextcloud/24940/htdocs/apps/encryption/lib/KeyManager.php line 404
    OCA\Encryption\Crypto\Crypt->decryptPrivateKey("*** sensitive parameters replaced ***")
  3. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Hooks/UserHooks.php line 183
    OCA\Encryption\KeyManager->init("*** sensitive parameter replaced ", " sensitive parameter replaced ***")
  4. /snap/nextcloud/24940/htdocs/lib/private/legacy/OC_Hook.php line 110
    OCA\Encryption\Hooks\UserHooks->login("*** sensitive parameters replaced ***")
  5. /snap/nextcloud/24940/htdocs/lib/private/Server.php line 581
    OC_Hook::emit("OC_User", "post_login", "*** sensitive parameter replaced ***")
  6. <>
    OC\Server->OC{closure}("*** sensitive parameters replaced ***")
  7. /snap/nextcloud/24940/htdocs/lib/private/Hooks/EmitterTrait.php line 101
    call_user_func_array(Closure {}, ["*** sensitive ... "])
  8. /snap/nextcloud/24940/htdocs/lib/private/Hooks/PublicEmitter.php line 40
    OC\Hooks\BasicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  9. /snap/nextcloud/24940/htdocs/lib/private/User/Session.php line 903
    OC\Hooks\PublicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  10. /snap/nextcloud/24940/htdocs/lib/base.php line 1058
    OC\User\Session->loginWithCookie("*** sensitive parameters replaced ***")
  11. /snap/nextcloud/24940/htdocs/lib/base.php line 997
    OC::handleLogin(OC\AppFramework\Http\Request {})
  12. /snap/nextcloud/24940/htdocs/index.php line 37
    OC::handleRequest()

Can I ask, which version of snap are you on. I'm on the latest stable version and re-enabling the CODE server and Collabora Online after updating the app had no positive impact at all, but just brought back the same problem.

@calbav
Copy link

calbav commented Dec 23, 2020

Can I ask, which version of snap are you on. I'm on the latest stable version and re-enabling the CODE server and Collabora Online after updating the app had no positive impact at all, but just brought back the same problem.

Hi, sure. Here is the output from snap list:

Name Version Rev Tracking Publisher Notes
nextcloud 20.0.3snap1 24940 latest/stable nextcloud* -

@SimplyCorbett
Copy link

SimplyCorbett commented Dec 23, 2020

Can you test on 20.0.4 instead of 20.0.3 @calbav

@calbav
Copy link

calbav commented Dec 23, 2020

@SimplyCorbett gladly, can you tell me how?

Would that be:

$ sudo snap refresh nextcloud --channel=<what channel? latest/beta?>

@theoneandonly-vector
Copy link

For me this doesn't happen on Rapberry Pi4 but on a x86vm only (same version).

@moshemalawach
Copy link

moshemalawach commented Jan 2, 2021

Happening on a latest version of docker images (x86_64), both apache and fpm... on a new install.

Was banging my head against walls for hours until I found that thread and disabled both apps (that are installed by default!)
Now it works perfectly...

@tyl0r
Copy link

tyl0r commented Jan 6, 2021

I just did a fresh NC install and noticed the same issue, which is what brought me here. I tried installing 20.0.1, 20.0.0, and 19.0.6 and they all had the same issue. I'm currently trying to get my company to use this software and really needed to evaluate NextCloud with Collabora. The issue appears to be isolated to the later versions of the apps (richdocuments and/or richdocumentscode). Here's what I did to get it working:

First, I installed the latest 20.0.4, again. I then downloaded older versions of richdocuments and richdocumentscode apps, extracted them to the nextcloud apps dir, and chown'd the dirs.

cd [your_nextcloud_dir]/apps
rm -r richdocuments/ richdocumentscode/
wget https://github.com/CollaboraOnline/richdocumentscode/releases/download/6.4.13/richdocumentscode.tar.gz
wget https://github.com/nextcloud/richdocuments/releases/download/v3.7.6/richdocuments.tar.gz
tar xzf richdocuments.tar.gz
tar xzf richdocumentscode.tar.gz 
chown -R www-data:www-data richdocuments*
rm -r *.tar.gz

Then I updated the database with the same versions of the apps:

update oc_appconfig set configvalue = '3.7.6' where appid = 'richdocuments' and configKey = 'installed_version';
update oc_appconfig set configvalue = '6.4.13' where appid = 'richdocumentscode' and configKey = 'installed_version';

Ultimately, I ended up disabling richdocumentscode and setup a standalone server for that.

update oc_appconfig set configvalue = 'no' where configkey = 'enabled' and appid = 'richdocumentscode';
select * from oc_appconfig where appid like 'richdocuments%';
+-------------------+----------------------------------+------------------------------------------+
| appid             | configkey                        | configvalue                              |
+-------------------+----------------------------------+------------------------------------------+
| richdocuments     | disable_certificate_verification |                                          |
| richdocuments     | enabled                          | yes                                      |
| richdocuments     | installed_version                | 3.7.6                                    |
| richdocuments     | public_wopi_url                  | https://collabora.clearhello.com         |
| richdocuments     | types                            | filesystem,dav,prevent_group_restriction |
| richdocuments     | wopi_url                         | https://collabora.clearhello.com         |
| richdocumentscode | enabled                          | no                                       |
| richdocumentscode | installed_version                | 6.4.13                                   |
| richdocumentscode | types                            |                                          |
+-------------------+----------------------------------+------------------------------------------+

Hope that helps someone!

@juliushaertl
Copy link
Member

I can finally reproduce the issue on one of my tests systems. I'll look into that.

@juliushaertl
Copy link
Member

There are basically two things we need to achieve here:

  • If no memcache is enabled the discovery/capabilities endpoints will get requested with each request of Nextcloud after Move Collabora endpoint caching to distributed cache #1226
    • We probably should reintroduce the old file-based caching layer as a fallback, however everyone should at least setup APCu caching for their Nextcloud instances as this helps a lot with the general performance
  • We should only request the richdocumentscode proxy if we get a proper state from the status endpoint
    • We might need some additional frontend code to make sure that we can attempt to open files even if the proxy is just starting and therefore the capabilities could not been fetched on page load yet
  • We should again lower the default timeout of requests if we are not going against the richdocumentscode proxy

@SimplyCorbett
Copy link

* however everyone should at least setup APCu caching for their Nextcloud instances as this helps a lot with the general performance.

I have no performance issues and no reason to setup APCu or redis.

If APCu will fix this issue I'll install it but I don't consider that a fix to the problem either.

@skydns
Copy link

skydns commented Jan 8, 2021

I'm manually downloading richdocumentscode_arm64.tar.gz 6.4.301 released 5h ago, it's a big package an github is slow so the download will be ready in 8h :(

I installed php-apcu and changed in nextcloud config.php line
'memcache.local' => '\OC\Memcache\Redis',
to
'memcache.local' => '\OC\Memcache\APCu',
after
snap restart nextcloud
my changes rewrite to Redis

I'll try tomorrow, if it will solve the perfomance problems.

@kyrofa
Copy link
Member

kyrofa commented Jan 8, 2021

I installed php-apcu and changed in nextcloud config.php line
'memcache.local' => '\OC\Memcache\Redis',
to
'memcache.local' => '\OC\Memcache\APCu',

I believe @juliushaertl was simply pointing out that there is an issue today if you don't have a memcache, not trying to say that you must use APCu. You have a memcache: it's redis.

@skydns
Copy link

skydns commented Jan 8, 2021

Thanks kyrofa, you're right. Tried with richdocumentscode_arm64 6.4.203
same issue

@pallebone
Copy link

I did a "sudo apt-get instal php-redis" and then "sudo snap restart nextcloud" but it did not resolve the issue. However I believe differently to SimplyCorbett that if this worked it would not be a solution. If it improves performance and allows richdocumentscode to work as it expects, this solution would be quite elegant and make the entire process more quick and snappy. Im not sure if there are some extra steps I missed to make it work.

@kyrofa
Copy link
Member

kyrofa commented Jan 8, 2021

"sudo apt-get instal php-redis" and then "sudo snap restart nextcloud"

Snap users, please stop doing this, you already have a memcache and this will not help anything.

@pallebone
Copy link

Ok sorry Im still learning and doing my best :)

@avimanyu786
Copy link

Gave up and disabled Collabora Online and Collabora Online - Built-in CODE Server. Installed OPENOFFICE and Community Document Server from the App store instead and then did the upgrade from 20.0.2 to 20.0.4. No idea why the community server for open office has a 1 star rating. Seems fine on my end.

@gerd21
Copy link

gerd21 commented Jan 11, 2021

Perhaps nor really necessary to state that I see this too. I use it in a VM with docker without root rights, I'm in the docker group. And I use 20.0.4, the latest docker image.
The interesting thing and the reason for my post is, that everything works fine, if I connect to NC on the host machine with 'http://localhost:8080'. Everything runs fast and in a way I expect.
BUT: this 'localhost' has has an IP as well. And if I try to enter that way 'http://192.168.178.34:8080' (for example) it is slow as described. Config.php is modified in trusted_hosts. And while the real IP is spinning, the parallel localhost tab works fine.
The 'fix' mentioned above works for me.

Perhaps it helps.

@vawaver
Copy link

vawaver commented Jan 11, 2021

Gave up and disabled Collabora Online and Collabora Online - Built-in CODE Server. Installed OPENOFFICE and Community Document Server from the App store instead and then did the upgrade from 20.0.2 to 20.0.4. No idea why the community server for open office has a 1 star rating. Seems fine on my end.

I have Ubuntu 20.04 server + Nextcloud via SNAP. Is there any tutorial on how to install Openoffice and Community Document Server? I cannot wait until the Collabora issue will be fixed. Meantime I would like to use another service for online editing of files.
Thank you.

@nextcloud nextcloud locked as off-topic and limited conversation to collaborators Jan 11, 2021
@juliushaertl
Copy link
Member

Please head over to the forum for discussing setup questions. I'll lock this conversation since we already spotted the actual issue and just need to dive into a solution as described in #1282 (comment)

@juliushaertl juliushaertl pinned this issue Jan 11, 2021
@juliushaertl juliushaertl changed the title RichDocuments causes significant slowdown/timeout issues in Nextcloud 20.0.2 RichDocuments built-in CODE causes significant slowdown/timeout issues Jan 11, 2021
@szaimen
Copy link
Contributor

szaimen commented Jun 8, 2022

@juliushaertl isn't this solved in the meantime? :)

@juliushaertl
Copy link
Member

No, there are still some scenarios where the requests to the code server may run longer and therefore block nextcloud requests in general. I've made an attempt to refactor the way that the connection is handled in #1901 I'll see if I can finish this soon.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
2. developing Work in progress bug Something isn't working high integration: built-in code server richdocumentscode performance
Projects
Status: 👀 In review