Error when trying to load channel with 30k messages

[mendelgordon]

Making a get request to the /api/channels/:name/messages route is failing with a 500 status code.
The message I get back is "Executor error during find command :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting."
I tried adding .allowDiskUse(true) to the function in server/api/channels/[name]/messages.get.ts but that didn't seem to help.

Error in terminal

[9:27:17 PM] [nuxt] [request error] [unhandled] [500] Executor error during find command :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.
at Connection.onMessage (./node_modules/mongodb/lib/cmap/connection.js:202:26)
at MessageStream. (./node_modules/mongodb/lib/cmap/connection.js:61:60)
at MessageStream.emit (node:events:514:28)
at processIncomingData (./node_modules/mongodb/lib/cmap/message_stream.js:124:16)
at MessageStream._write (./node_modules/mongodb/lib/cmap/message_stream.js:33:9)
at writeOrBuffer (node:internal/streams/writable:399:12)
at _write (node:internal/streams/writable:340:10)
at Writable.write (node:internal/streams/writable:344:10)
at TLSSocket.ondata (node:internal/streams/readable:785:22)
at TLSSocket.emit (node:events:514:28)
[9:27:18 PM] [nuxt] [request error] [unhandled] [500] Executor error during find command :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.
at Connection.onMessage (./node_modules/mongodb/lib/cmap/connection.js:202:26)
at MessageStream. (./node_modules/mongodb/lib/cmap/connection.js:61:60)
at MessageStream.emit (node:events:514:28)
at processIncomingData (./node_modules/mongodb/lib/cmap/message_stream.js:124:16)
at MessageStream._write (./node_modules/mongodb/lib/cmap/message_stream.js:33:9)
at writeOrBuffer (node:internal/streams/writable:399:12)
at _write (node:internal/streams/writable:340:10)
at Writable.write (node:internal/streams/writable:344:10)
at TLSSocket.ondata (node:internal/streams/readable:785:22)
at TLSSocket.emit (node:events:514:28)

[mendelgordon]

Clicking on the channel in the channels dropdown triggers this issue - the page just says This channel is empty.
The messages are definitely in the database since when I search I see them, but pressing on them to load them in the channel goes to the same This channel is empty page.

[mendelgordon]

I got it to work when using a local mongo db - issue only occurs when I'm using the free cloud option that mongodb provides.
Now the question is how to add pagination or infinite scroll so it doesn't lag as much 😄

[4350pChris]

Hey, thanks for your efforts!
The basic problem is that I don't seem to know enough about MongoDB to do the work of fetching the messages directly from the DB in the way I need them, that is have replies nested under the comment they reply to. That's why there's normalizeMessages in the backend - it does just that.

And the problem with pagination is, that it doesn't circumvent this issue as I can't just fetch X messages from the DB since e.g. if I fetch messages 1000 - 2000 in a channel of 10k messages then a reply to one of those 1000 messages I fetched might very well be outside that range and therefore missing.

I think there's two options:

1. Fetch X messages (pagination), then query the DB again to find all replies to these messages.
2. Find out how to do all of this in a single query (preferable)

If you feel like having a go at this it'd be much appreciated. I don't have the time right now, so I don't know when / if ever I'll implement this.