Hanging database - indexer waiting for transactions, query waiting for indexer

[jarohen]

From @nivekuil on Clojurians

Details:

• in-mem tx-log, local-disk storage (haven't tried with local/local or in-mem/in-mem)

• seems to be a necessary condition that it's written at least a block to disk (i.e. starting from a clean tx-log and storage doesn't repro this)

• then pmap 50 docs (pmap seems to be a necessary condition)

  (pmap (fn [x]
          (xt/submit-tx node [[:put-docs :test {:xt/id (random-uuid)}]]))
        (repeat 50 1))

• further queries then hang - the transaction indexing thread awaiting further transactions from the log, the query awaiting transactions to be indexed.

  "xtdb-tx-subscription-pool-1-thread-2" #128 [2042437] prio=5 os_prio=0 cpu=0.15ms elapsed=84.41s tid=0x0000739c0cf185a0 nid=2042437 waiting on condition  [0x0000739f0cdcc000]
    java.lang.Thread.State: WAITING (parking)
        [...]
        at java.util.concurrent.Semaphore.acquire(java.base@22/Semaphore.java:318)
        at xtdb.log.NotifyingSubscriberHandler$fn__74636.invoke(log.clj:106)
        [...]
  
  "nREPL-session-a1361af3-7e3a-4488-abcc-65364b4bbee9" #252 [2044496] daemon prio=5 os_prio=0 cpu=17.34ms elapsed=453.42s tid=0x0000739cc00012f0 nid=2044496 waiting on condition  [0x0000739f0e5e3000]
    java.lang.Thread.State: WAITING (parking)
        [...]
        at clojure.core$deref.invoke(core.clj:2323)
        at xtdb.api$q.invokeStatic(api.clj:159)
        [...]
  

• latest-completed-tx is behind latest-submitted-tx. Seems to be plenty of transactions available through the log - my (unsubstantiated) gut feel is that it's a race condition in the in-memory tx-log transaction notification?

• This also may well be an issue caused by an in-memory tx-log vs a local-disk storage, with the former being reset without the latter.