Fix mongo OOM during sync #42140

qnkhuat · 2024-05-02T08:26:18Z

The big memory usage comes from the mongo.util/do-find method, which return a vector. We should use a lazy seq instead.

This is a regression and introduced by #38017

Here is a script to demonstrate memory usage before and after the changes.

(require 
 '[toucan2.core :as t2]
 '[metabase.util.log :as log]
 '[metabase.sync.sync-metadata.fields :as sync.fields])

(def db (t2/select-one :model/Database :engine :mongo))

(future
 (def max-memory-usage (atom 0))
 (doseq [_ (range 100)]
   (let [memory-usage (float (/ (- (.. Runtime getRuntime totalMemory) (.. Runtime getRuntime freeMemory)) 1024 1024))]
     (swap! max-memory-usage max memory-usage)
     (log/info (format "Memory usage: %f MB, Max memory usage: %f MB" memory-usage @max-memory-usage)))
   (Thread/sleep 500)))

(do
  (log/info "Start")
  (sync.fields/sync-fields! db)
  (log/info "Done"))

Before:

2024-05-02 08:19:48,656 INFO local.mongo-oom-sync-fields :: Start
2024-05-02 08:19:48,982 INFO local.mongo-oom-sync-fields :: Memory usage: 1012.117493 MB, Max memory usage: 1404.911377 MB
2024-05-02 08:19:49,483 INFO local.mongo-oom-sync-fields :: Memory usage: 1901.464355 MB, Max memory usage: 1901.464355 MB
2024-05-02 08:19:49,986 INFO local.mongo-oom-sync-fields :: Memory usage: 2566.096924 MB, Max memory usage: 2566.096924 MB
2024-05-02 08:19:50,491 INFO local.mongo-oom-sync-fields :: Memory usage: 2572.239014 MB, Max memory usage: 2572.239014 MB
2024-05-02 08:19:50,993 INFO local.mongo-oom-sync-fields :: Memory usage: 2578.385010 MB, Max memory usage: 2578.385010 MB
2024-05-02 08:19:51,497 INFO local.mongo-oom-sync-fields :: Memory usage: 2584.527100 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:52,002 INFO local.mongo-oom-sync-fields :: Memory usage: 2409.627441 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:52,508 INFO local.mongo-oom-sync-fields :: Memory usage: 2415.769531 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:53,009 INFO local.mongo-oom-sync-fields :: Memory usage: 2421.915527 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:53,512 INFO local.mongo-oom-sync-fields :: Memory usage: 2428.057617 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:54,017 INFO local.mongo-oom-sync-fields :: Memory usage: 2434.203613 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:54,519 INFO local.mongo-oom-sync-fields :: Memory usage: 2440.345459 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:55,023 INFO local.mongo-oom-sync-fields :: Memory usage: 2446.491455 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:55,525 INFO local.mongo-oom-sync-fields :: Memory usage: 2452.633545 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:56,030 INFO local.mongo-oom-sync-fields :: Memory usage: 2458.779541 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:56,531 INFO local.mongo-oom-sync-fields :: Memory usage: 2464.921387 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:57,036 INFO local.mongo-oom-sync-fields :: Memory usage: 2471.067383 MB, Max memory usage: 2584.527100 MB
2024-05-02 08:19:57,323 INFO local.mongo-oom-sync-fields :: Done

After

2024-05-02 08:18:48,387 INFO local.mongo-oom-sync-fields :: Start
2024-05-02 08:18:48,656 INFO local.mongo-oom-sync-fields :: Memory usage: 1933.911987 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:49,157 INFO local.mongo-oom-sync-fields :: Memory usage: 891.292480 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:49,662 INFO local.mongo-oom-sync-fields :: Memory usage: 1117.596191 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:50,167 INFO local.mongo-oom-sync-fields :: Memory usage: 1250.718262 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:50,672 INFO local.mongo-oom-sync-fields :: Memory usage: 1478.044189 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:51,177 INFO local.mongo-oom-sync-fields :: Memory usage: 1705.374268 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:51,678 INFO local.mongo-oom-sync-fields :: Memory usage: 1932.700195 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:52,179 INFO local.mongo-oom-sync-fields :: Memory usage: 748.504089 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:52,681 INFO local.mongo-oom-sync-fields :: Memory usage: 975.960144 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:53,186 INFO local.mongo-oom-sync-fields :: Memory usage: 1158.104126 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:53,688 INFO local.mongo-oom-sync-fields :: Memory usage: 1385.434082 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:54,189 INFO local.mongo-oom-sync-fields :: Memory usage: 1612.760132 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:54,694 INFO local.mongo-oom-sync-fields :: Memory usage: 1745.882080 MB, Max memory usage: 1933.911987 MB
2024-05-02 08:18:55,200 INFO local.mongo-oom-sync-fields :: Memory usage: 1971.162109 MB, Max memory usage: 1971.162109 MB
2024-05-02 08:18:55,704 INFO local.mongo-oom-sync-fields :: Memory usage: 847.578308 MB, Max memory usage: 1971.162109 MB
2024-05-02 08:18:56,205 INFO local.mongo-oom-sync-fields :: Memory usage: 1042.651367 MB, Max memory usage: 1971.162109 MB
2024-05-02 08:18:56,710 INFO local.mongo-oom-sync-fields :: Memory usage: 1208.026245 MB, Max memory usage: 1971.162109 MB
2024-05-02 08:18:57,215 INFO local.mongo-oom-sync-fields :: Memory usage: 1435.356323 MB, Max memory usage: 1971.162109 MB
2024-05-02 08:18:57,518 INFO local.mongo-oom-sync-fields :: Done

You can see that using lazy seq the memory usage barely increases at all.

qnkhuat · 2024-05-02T08:34:36Z

modules/drivers/mongo/src/metabase/driver/mongo/util.clj

@@ -70,7 +70,7 @@
 skip (.skip skip)
 batch-size (.batchSize (int batch-size))
 sort-criteria (.sort (mongo.conversion/to-document (ordered-map/ordered-map sort-criteria))))
- (mapv #(mongo.conversion/from-document % opts))))
+ (map #(mongo.conversion/from-document % opts))))


there are more places here that use mapv. My gut says we should convert them all. Should we? @lbrdnk

I'll open tech debt issue tracking this.

qnkhuat · 2024-05-02T08:36:28Z

modules/drivers/mongo/test/metabase/driver/mongo/util_test.clj

I noticed we don't have any behavior test here. I can add it later if we want to. but this PR should fix the escalation only.

replay-io · 2024-05-02T08:38:11Z

Status	In Progress ↗︎ 53 / 54
Commit	`0201a6a`
Results	⚠️ 8 Flaky ✅ 2444 Passed

lbrdnk

LGTM.

lbrdnk · 2024-05-02T09:21:02Z

modules/drivers/mongo/src/metabase/driver/mongo/util.clj

@@ -70,7 +70,7 @@
 skip (.skip skip)
 batch-size (.batchSize (int batch-size))
 sort-criteria (.sort (mongo.conversion/to-document (ordered-map/ordered-map sort-criteria))))
- (mapv #(mongo.conversion/from-document % opts))))
+ (map #(mongo.conversion/from-document % opts))))


I'll open tech debt issue tracking this.

github-actions · 2024-05-02T12:16:55Z

@qnkhuat Did you forget to add a milestone to the issue for this PR? When and where should I add a milestone?

Co-authored-by: Ngoc Khuat <[email protected]>

crisptrutski · 2024-05-07T11:56:38Z

modules/drivers/mongo/test/metabase/driver/mongo/util_test.clj

+ (is (= clojure.lang.LazySeq
+ (type (mongo.util/do-find (mongo.util/collection db "venues")))))))))


Rather than coding to a concrete type, and assuming that it hasn't been realized, you could instead assert that realized? is false.

oh yea that'd be nicer.

Fix mongo OOM during sync

4e33336

qnkhuat requested a review from camsaul as a code owner May 2, 2024 08:26

metabase-bot bot assigned qnkhuat May 2, 2024

metabase-bot bot added the .Team/BackendComponents also known as BEC label May 2, 2024

qnkhuat added the backport Automatically create PR on current release branch on merge label May 2, 2024

qnkhuat requested review from lbrdnk and a team May 2, 2024 08:27

qnkhuat commented May 2, 2024

View reviewed changes

lbrdnk approved these changes May 2, 2024

View reviewed changes

qnkhuat enabled auto-merge (squash) May 2, 2024 09:24

Merge branch 'master' into fix-mongo-oom

0201a6a

lbrdnk mentioned this pull request May 2, 2024

Consider changing calls to mapv into map in mongo conversions code #42147

Open

qnkhuat merged commit 3db638f into master May 2, 2024
108 checks passed

qnkhuat deleted the fix-mongo-oom branch May 2, 2024 12:15

github-actions bot pushed a commit that referenced this pull request May 2, 2024

Fix mongo OOM during sync (#42140)

177d90c

metabase-bot bot mentioned this pull request May 2, 2024

🤖 backported "Fix mongo OOM during sync" #42152

Merged

metabase-bot bot added a commit that referenced this pull request May 2, 2024

Fix mongo OOM during sync (#42140) (#42152)

38cb850

Co-authored-by: Ngoc Khuat <[email protected]>

qnkhuat added this to the 0.49.9 milestone May 3, 2024

crisptrutski reviewed May 7, 2024

View reviewed changes

sloansparger modified the milestones: 0.49.9, 0.49.8 May 8, 2024

dosubot bot modified the milestone: 0.49.8 May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix mongo OOM during sync #42140

Fix mongo OOM during sync #42140

qnkhuat commented May 2, 2024 •

edited

qnkhuat May 2, 2024

lbrdnk May 2, 2024

qnkhuat May 2, 2024

replay-io bot commented May 2, 2024 •

edited

lbrdnk left a comment

lbrdnk May 2, 2024

github-actions bot commented May 2, 2024

crisptrutski May 7, 2024

qnkhuat May 7, 2024

		(is (= clojure.lang.LazySeq
		(type (mongo.util/do-find (mongo.util/collection db "venues")))))))))

Fix mongo OOM during sync #42140

Fix mongo OOM during sync #42140

Conversation

qnkhuat commented May 2, 2024 • edited

qnkhuat May 2, 2024

Choose a reason for hiding this comment

lbrdnk May 2, 2024

Choose a reason for hiding this comment

qnkhuat May 2, 2024

Choose a reason for hiding this comment

replay-io bot commented May 2, 2024 • edited

lbrdnk left a comment

Choose a reason for hiding this comment

lbrdnk May 2, 2024

Choose a reason for hiding this comment

github-actions bot commented May 2, 2024

crisptrutski May 7, 2024

Choose a reason for hiding this comment

qnkhuat May 7, 2024

Choose a reason for hiding this comment

qnkhuat commented May 2, 2024 •

edited

replay-io bot commented May 2, 2024 •

edited