regopy is seemingly incompatible with python-greenlet

[nazywam]

This is a short summary of a long battle with weird issues trying to integrate regopy into a project that (indirectly) uses greenlets.

Bug reproduction steps:

virtualenv -ppython3 venv
source venv/bin/activate.fish
pip install regopy==0.4.5 greenlet==3.1.0
python3 -c "import greenlet; import regopy"

python version used: Python 3.11.6

output:

free(): invalid pointer
fish: Job 1, 'python3 -c "import greenlet; im…' terminated by signal SIGABRT (Abort)

full stacktrace:

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c428e6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c268b7 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c27750 in __libc_message (fmt=fmt@entry=0x7ffff7dc3b34 "%s\n") at ../sysdeps/posix/libc_fatal.c:150
#6  0x00007ffff7ca3ce7 in malloc_printerr (str=str@entry=0x7ffff7dc1612 "free(): invalid pointer") at ./malloc/malloc.c:5765
#7  0x00007ffff7ca5f24 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at ./malloc/malloc.c:4500
#8  0x00007ffff7ca8873 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9  0x00007ffff6f00056 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff6f0044f in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(unsigned long, char) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007ffff6ae76bf in ?? () from /tmp/test-case/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-x86_64-linux-gnu.so
#12 0x00007ffff6ae7799 in date::set_install(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /tmp/test-case/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-x86_64-linux-gnu.so
#13 0x00007ffff67692a4 in rego::set_tzdata_path(std::filesystem::path const&) ()
   from /tmp/test-case/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-x86_64-linux-gnu.so
#14 0x00007ffff674db82 in regoSetTZDataPath () from /tmp/test-case/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-x86_64-linux-gnu.so
#15 0x00007ffff674677d in ?? () from /tmp/test-case/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-x86_64-linux-gnu.so
#16 0x00007ffff6741f34 in ?? () from /tmp/test-case/venv/lib/python3.11/site-packages/regopy/_regopy.cpython-311-x86_64-linux-gnu.so
#17 0x00000000005124e6 in cfunction_call (func=0x7ffff7299440, args=<optimized out>, kwargs=<optimized out>) at ../Objects/methodobject.c:542
#18 0x00000000004e08cb in _PyObject_MakeTpCall (tstate=0xa71a18 <_PyRuntime+166328>, callable=0x7ffff7299440, args=0x7ffff7fb33c8, nargs=<optimized out>, keywords=0x0)
    at ../Objects/call.c:214
#19 0x00000000004f5f7a in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4760
#20 0x00000000005d9fdf in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb3370, tstate=0xa71a18 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#21 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=<optimized out>, func=0x7ffff726e700, tstate=0xa71a18 <_PyRuntime+166328>) at ../Python/ceval.c:6425
#22 PyEval_EvalCode (co=0xb418e0, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:1140
#23 0x00000000005f21b4 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7ffff73f3340, globals=0x7ffff73f3340, source=0xb418e0)
    at ../Python/bltinmodule.c:1077
#24 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=2, kwnames=<optimized out>) at ../Python/clinic/bltinmodule.c.h:465
#25 0x00000000005056d7 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7ffff7388fe0, args=0x7ffff72891d8, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Include/cpython/methodobject.h:52
#26 0x00000000004fe2c1 in do_call_core (use_tracing=<optimized out>, kwdict=0x7ffff73fff40, callargs=0x7ffff72891c0, func=0x7ffff7388fe0, tstate=<optimized out>)
    at ../Python/ceval.c:7315
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:5367
#28 0x0000000000525bf5 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb3078, tstate=0xa71a18 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#29 _PyEval_Vector (kwnames=<optimized out>, argcount=2, args=0x7fffffffdf00, locals=0x0, func=0x7ffff734fce0, tstate=0xa71a18 <_PyRuntime+166328>) at ../Python/ceval.c:6425
#30 _PyFunction_Vectorcall (func=0x7ffff734fce0, stack=0x7fffffffdf00, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:393
#31 0x0000000000510d38 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffffdf00, callable=0x7ffff734fce0, tstate=0xa71a18 <_PyRuntime+166328>)
    at ../Include/internal/pycore_call.h:92
#32 object_vacall (tstate=0xa71a18 <_PyRuntime+166328>, base=<optimized out>, callable=0x7ffff734fce0, vargs=0x7fffffffdf90) at ../Objects/call.c:819
#33 0x000000000054cc4f in PyObject_CallMethodObjArgs (obj=0x0, name=<optimized out>) at ../Objects/call.c:878
#34 0x000000000054b932 in import_find_and_load (abs_name=0x7ffff727e770, tstate=0xa71a18 <_PyRuntime+166328>) at ../Python/import.c:1737
#35 PyImport_ImportModuleLevelObject (name=0x7ffff727e770, globals=<optimized out>, locals=<optimized out>, fromlist=0x946cc0 <_Py_NoneStruct>, level=0)
    at ../Python/import.c:1836
#36 0x00000000004fab73 in import_name (level=0xa493a8 <_PyRuntime+840>, fromlist=0x946cc0 <_Py_NoneStruct>, name=0x7ffff727e770, frame=<optimized out>,
    tstate=<optimized out>) at ../Python/ceval.c:7415
#37 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3937
#38 0x00000000005d9fdf in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb3020, tstate=0xa71a18 <_PyRuntime+166328>) at ../Include/internal/pycore_ceval.h:73
#39 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=<optimized out>, func=0x7ffff73d1f80, tstate=0xa71a18 <_PyRuntime+166328>) at ../Python/ceval.c:6425
#40 PyEval_EvalCode (co=0x7ffff724c5e0, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:1140
#41 0x00000000005f8687 in run_eval_code_obj (tstate=0xa71a18 <_PyRuntime+166328>, co=0x7ffff724c5e0, globals=0x7ffff73f3240, locals=0x7ffff73f3240)

I think issue comes down to:

• regoSetTZDataPath(os.path.join(os.path.basename(__file__), "tzdata")) -

  rego-cpp/wrappers/python/src/regopy/__init__.py

  Line 36 in 756a920

  regoSetTZDataPath(os.path.join(os.path.basename(__file__), "tzdata"))

• date::set_install(path.string()); -

  rego-cpp/src/builtins/time.cc

  Line 748 in 756a920

  date::set_install(path.string());

• set_install(const std::string& install) - https://github.com/HowardHinnant/date/blob/1a4f424659d39c2a222729bd2b1ccd8f857b3221/src/tz.cpp#L443

The order in which the modules are imported does matter, which makes me think that greenlet initialization does something that date does not like.

The issue seems to come up only in greenlet>=2.0.0 and regopy>0.3.11

[matajoh]

Thanks for the issue, and for the extensive investigation! I will look into this and see what I can find. I anticipated the date code may result in some unexpected side effects, but nothing quite like this.