[BUG] nvtabular backend seg faults on triton termination

[jperez999]

when shutting down triton get the following error if nvtabular backend used:

I0608 15:23:57.627423 7128 server.cc:295] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I0608 15:23:57.627598 7128 backend.cc:160] TRITONBACKEND_ModelInstanceFinalize: delete instance state
Signal (11) received.
0# 0x0000555E4A98E1B9 in tritonserver
1# 0x00007F00122130C0 in /usr/lib/x86_64-linux-gnu/libc.so.6
2# 0x00007EFFEDCF2F3E in /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
3# TRITONBACKEND_ModelInstanceFinalize in /opt/tritonserver/backends/nvtabular/libtriton_nvtabular.so
4# 0x00007F0012ABF1A2 in /opt/tritonserver/lib/libtritonserver.so
5# 0x00007F0012AB4E79 in /opt/tritonserver/lib/libtritonserver.so
6# 0x00007F0012AB560D in /opt/tritonserver/lib/libtritonserver.so
7# 0x00007F0012B5C687 in /opt/tritonserver/lib/libtritonserver.so
8# 0x00007F0012604DE4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
9# 0x00007F0013812609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0
10# clone in /usr/lib/x86_64-linux-gnu/libc.so.6

Segmentation fault (core dumped)

causes tests to hang when running in pytest, cannot shutdown tritonserver correctly.

when running the test_triton_inference.py::test_seq_etl_tf_model

[viswa-nvidia]

@jperez999 @karlhigley , please update the workaround that's in place in this ticket

[karlhigley]

The workaround is to use the Triton Python back-end instead of our custom NVT back-end (which is a stripped down version of the Python back-end anyway.)