Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

restart with export DX_CLEAN=false may fail #84

Open
HAOYUatHZ opened this issue Sep 23, 2021 · 1 comment
Open

restart with export DX_CLEAN=false may fail #84

HAOYUatHZ opened this issue Sep 23, 2021 · 1 comment

Comments

@HAOYUatHZ
Copy link
Member

description

restart with export DX_CLEAN=false may fail for

  • prover-cluster client
  • block_submitter
  • tick
  • restapi
  • contracts ticker

how to reproduce

export DX_CLEAN=false
./stopsh
./run.sh

log

ubuntu@ip-172-31-18-140:~/repos/fluidex-backend$ make tail_log 
docker-compose --file orchestra/docker/docker-compose.yaml --project-name orchestra logs > orchestra/docker-compose.log
docker-compose --file regnbue-bridge/docker/docker-compose.yaml --project-name faucet logs > regnbue-bridge/docker-compose.log
ls rollup-state-manager/*.log prover-cluster/*.log dingir-exchange/*.log dingir-exchange/logs/*.log regnbue-bridge/*.log contracts/*.log orchestra/*.log | xargs tail -n 3
==> contracts/ganache.2021-09-23.log <==
9007199254740991

Listening on 127.0.0.1:8545

==> contracts/ticker.2021-09-23.log <==
        at emitErrorNT (node:internal/streams/destroy:193:8)
        at emitErrorCloseNT (node:internal/streams/destroy:158:3)
        at processTicksAndRejections (node:internal/process/task_queues:83:21)

==> dingir-exchange/logs/matchengine.2021-09-23.log <==
    
Sep 23 04:10:38.953  INFO dingir_exchange::message::producer: kafka producer disconnected    
Sep 23 04:10:38.953  INFO dingir_exchange::message::producer: kafka producer running terminated    

==> dingir-exchange/logs/persistor.2021-09-23.log <==
Sep 23 04:10:40.281  INFO dingir_exchange::message::persist: start auto commiting for topic internaltransfer    
Sep 23 04:10:40.281  INFO dingir_exchange::message::persist: start auto commiting for topic registeruser    
Sep 23 04:10:41.301  INFO dingir_exchange::message::consumer: start consuming topic ["trades", "internaltransfer", "registeruser", "balances", "orders"]    

==> dingir-exchange/logs/restapi.2021-09-23.log <==
  44: __libc_start_main
  45: _start
    

==> dingir-exchange/tick.2021-09-23.log <==
  details: 'No connection established',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

==> orchestra/docker-compose.log <==
exchange_envoy        | [2021-09-23 04:19:48.260][19][debug][connection] [source/common/network/connection_impl.cc:243] [C49] closing socket: 0
exchange_envoy        | [2021-09-23 04:19:48.260][19][debug][client] [source/common/http/codec_client.cc:107] [C49] disconnect. resetting 0 pending requests
exchange_envoy        | [2021-09-23 04:19:48.260][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:407] [C49] client disconnected, failure reason: 

==> prover-cluster/client.2021-09-23.log <==
  15: __libc_start_main
  16: _start
    

==> prover-cluster/coordinator.2021-09-23.log <==
  status = $1
  and updated_time < current_timestamp - interval '172800 seconds'
  

==> regnbue-bridge/block_submitter.2021-09-23.log <==
    0: error trying to connect: tcp connect error: Connection refused (os error 111)
    1: tcp connect error: Connection refused (os error 111)
    2: Connection refused (os error 111)

==> regnbue-bridge/docker-compose.log <==
regnbue_bridge_pq | 2021-09-23 04:10:26.580 UTC [28] LOG:  TimescaleDB background worker launcher connected to shared catalogs
regnbue_bridge_pq | 2021-09-23 04:11:31.098 UTC [51] LOG:  the "timescaledb" extension is not up-to-date
regnbue_bridge_pq | 2021-09-23 04:11:31.098 UTC [51] HINT:  The most up-to-date version is 2.4.2, the installed version is 2.1.0.

==> regnbue-bridge/faucet.2021-09-23.log <==
  
Sep 23 04:19:50.229  INFO sqlx::query: COMMIT; rows: 0, elapsed: 93.348µs  
Sep 23 04:19:50.230  INFO sqlx::query: /* SQLx ping */; rows: 0, elapsed: 83.309µs  

==> rollup-state-manager/rollup_state_manager.2021-09-23.log <==
Sep 23 04:15:22.823  INFO rollup_state_manager: generate 0 blocks with block_size 2 in 240.00372s: average TPS: 0    
Sep 23 04:17:22.823  INFO rollup_state_manager: generate 0 blocks with block_size 2 in 360.00385s: average TPS: 0    
Sep 23 04:19:22.823  INFO rollup_state_manager: generate 0 blocks with block_size 2 in 480.00394s: average TPS: 0    
ubuntu@ip-172-31-18-140:~/repos/fluidex-backend$ tail -n 300 dingir-exchange/logs/restapi.2021-09-23.log
Sep 23 04:10:40.256 DEBUG restapi: Prepared DB connection: postgres://exchange:[email protected]/exchange    
Sep 23 04:10:40.257  INFO restapi: Connect to manage channel http://0.0.0.0:50051    
Sep 23 04:10:41.666 ERROR fluidex_common::non_blocking_tracing: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))', src/bin/restapi.rs:34:18
   0: fluidex_common::non_blocking_tracing::get_backtrace
             at /home/ubuntu/.cargo/git/checkouts/common-rs-f10c3d305ff6aa0a/24293b3/src/non_blocking_tracing.rs:50:30
   1: fluidex_common::non_blocking_tracing::set_panic_hook::{{closure}}
             at /home/ubuntu/.cargo/git/checkouts/common-rs-f10c3d305ff6aa0a/24293b3/src/non_blocking_tracing.rs:39:13
   2: std::panicking::rust_panic_with_hook
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:595:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:497:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/sys_common/backtrace.rs:141:18
   5: rust_begin_unwind
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:493:5
   6: core::panicking::panic_fmt
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/panicking.rs:92:14
   7: core::result::unwrap_failed
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/result.rs:1355:5
   8: core::result::Result<T,E>::unwrap
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/result.rs:1037:23
   9: restapi::main::{{closure}}
             at src/bin/restapi.rs:29:13
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/mod.rs:80:19
  11: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll::{{closure}}::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:668:65
  12: tokio::coop::with_budget::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:106:9
  13: std::thread::local::LocalKey<T>::try_with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:376:16
  14: std::thread::local::LocalKey<T>::with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:352:9
  15: tokio::coop::with_budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:99:5
      tokio::coop::budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:76:5
      <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:668:42
  16: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/macros/scoped_tls.rs:61:9
  17: tokio::task::local::LocalSet::with
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:573:9
  18: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:658:9
  19: tokio::task::local::LocalSet::run_until::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:516:9
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/mod.rs:80:19
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/future.rs:120:9
  22: tokio::runtime::basic_scheduler::Inner<P>::block_on::{{closure}}::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:208:62
  23: tokio::coop::with_budget::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:106:9
  24: std::thread::local::LocalKey<T>::try_with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:376:16
  25: std::thread::local::LocalKey<T>::with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:352:9
  26: tokio::coop::with_budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:99:5
      tokio::coop::budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:76:5
      tokio::runtime::basic_scheduler::Inner<P>::block_on::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:208:39
  27: tokio::runtime::basic_scheduler::enter::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:299:29
  28: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/macros/scoped_tls.rs:61:9
  29: tokio::runtime::basic_scheduler::enter
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:299:5
  30: tokio::runtime::basic_scheduler::Inner<P>::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:197:9
  31: tokio::runtime::basic_scheduler::InnerGuard<P>::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:452:9
  32: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:157:24
  33: tokio::runtime::Runtime::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/mod.rs:450:46
  34: tokio::task::local::LocalSet::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:477:9
  35: actix_rt::runtime::Runtime::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.1.0/src/runtime.rs:85:9
  36: actix_rt::system::SystemRunner::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.1.0/src/system.rs:186:9
  37: restapi::main
             at src/bin/restapi.rs:17:1
  38: core::ops::function::FnOnce::call_once
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops/function.rs:227:5
  39: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/sys_common/backtrace.rs:125:18
  40: std::rt::lang_start::{{closure}}
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:49:18
  41: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:379:40
      std::panicking::try
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:343:19
      std::panic::catch_unwind
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panic.rs:431:14
      std::rt::lang_start_internal
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:34:21
  42: std::rt::lang_start
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:48:5
  43: main
  44: __libc_start_main
  45: _start
@HAOYUatHZ
Copy link
Member Author

HAOYUatHZ commented Oct 27, 2021 •
edited
Loading

I try again but cannot reproduce.

This time the binaries can start but the data looks weird: kline lose old data & we don't have new blocks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@HAOYUatHZ