Segfaults with lz4mt (threaded mode) and deadlocks with Thread-Pool branch

[blappm]

Hi,

I'm trying to use your extemly fast version of lz4, but from time to time
lz4mt is segfaulting due to a issue within threading. It happens with both compression
and decompression, but not always at the same place.

We used gcc7 to compile it.

I've also tried the thread-pool variant, but there I see deadlocks - the compression is stalling.

Why haven't you merged the Thread-Pool veriant into the main tree? Was there
any reason do not to do it? It seems to me that it would fix the segfault problems.

Best Regards:
Martin

[blappm]

Crash with -O2. I'm currently trying to see if I can get crashes with -O0 too.

Core was generated by `lz4mt -1 --numthreads=3 -c'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000004111f9 in _S_checkstd::__future_base::_State_baseV2 (__p=...) at /usr/include/c++/7/future:552552
if (!static_cast(__p))

(gdb) where

#0 0x00000000004111f9 in _S_checkstd::__future_base::_State_baseV2 (__p=...) at /usr/include/c++/7/future:552#1
wait (this=0x7f37f3bfeff0) at /usr/include/c++/7/future:691#2
(anonymous namespace)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>::operator()(int, Lz4Mt::MemPool::Buffer , int) const (__closure=0x7f37f506e350, i=262143, srcRawPtr=,
srcSize=) at src/lz4mt.cpp:408#3
0x00000000004111f9 in lz4mtCompress (lz4MtContext=, sd=)
#4 0x0000000000412072 in __invoke_impl<void, (anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer, int)>, int, Lz4Mt::MemPool::Buffer*, int> (__f=) at /usr/include/c++/7/bits/invoke.h:60
#5 __invoke<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> (__fn=) at /usr/include/c++/7/bits/invoke.h:95
#6 _M_invoke<0, 1, 2, 3> (this=) at /usr/include/c++/7/thread:234#7
operator() (this=) at /usr/include/c++/7/thread:243#8
operator() (this=0x7f37fa1aee30) at /usr/include/c++/7/future:1362#9
std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >, void> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:302
#10 0x000000000040b9eb in operator() (this=) at /usr/include/c++/7/bits/std_function.h:706
#11 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x7f37f506e300,
__f=, __did_set=0x7f37fa1aee07) at /usr/include/c++/7/future:561#12
0x00007f37fc7b8d03 in pthread_once () from /lib64/libpthread.so.0
#13 0x000000000040ec48 in __gthread_once (__func=, __once=0x7f37f506e318) at /usr/include/c++/7/x86_64-suse-linux/bits/gthr-default.h:699
#14 call_once<void (std::__future_base::_State_baseV2::)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>, bool> (__f=<unknown type in /tmp/lz4mt.exe, CU 0x34048, DIE 0x6e524>,
__once=...) at /usr/include/c++/7/mutex:684#15
_M_set_result (__ignore_failure=false, __res=..., this=0x7f37f506e300) at /usr/include/c++/7/future:401#16
operator() (__closure=0x7f37f506e0e8) at /usr/include/c++/7/future:1667#17
__invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >; _Res = void]::<lambda()> > (__f=)
at /usr/include/c++/7/bits/invoke.h:60
#18 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >; _Res = void]::<lambda()> > (__fn=)
at /usr/include/c++/7/bits/invoke.h:95
#19 _M_invoke<0> (this=0x7f37f506e0e8) at /usr/include/c++/7/thread:234#20
operator() (this=0x7f37f506e0e8) at /usr/include/c++/7/thread:243#21
std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >; _Res = void]::<lambda()> > > >::_M_run(void) (this=0x7f37f506e0e0) at /usr/include/c++/7/thread:186#22
0x0000000000488cdf in std::execute_native_thread_routine (__p=0x7f37f506e0e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#23 0x00007f37fc7b3806 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f37fc2959bd in signalfd () from /lib64/libc.so.6
#25 0x0000000000000000 in ?? ()
(gdb) p __p
$1 = (const std::shared_ptrstd::__future_base::_State_baseV2 &) @0x7f37f3bfeff0: {<std::__shared_ptr<std::__future_base::_State_baseV2, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<std::__future_base::_State_baseV2, (__gnu_cxx::_Lock_policy)2, false, false>> = {}, _M_ptr = 0x7f37f506e270, _M_refcount = {_M_pi = 0x7f37f506e260}}, }