Can't pickle local object 'main.<locals>.grpc_prediction_server' #3410

minhdang241 · 2021-07-16T06:48:33Z

Describe the bug

I have follow the tutorial here and got the error as

2021-07-16 13:45:27,155 - seldon_core.microservice:main:203 - INFO:  Starting microservice.py:main
2021-07-16 13:45:27,155 - seldon_core.microservice:main:204 - INFO:  Seldon Core version: 1.9.1
2021-07-16 13:45:27,159 - seldon_core.microservice:main:345 - INFO:  Parse JAEGER_EXTRA_TAGS []
2021-07-16 13:45:27,160 - seldon_core.microservice:main:348 - INFO:  Annotations: {}
2021-07-16 13:45:27,160 - seldon_core.microservice:main:352 - INFO:  Importing MyModel
2021-07-16 13:45:28,526 - seldon_core.microservice:main:396 - INFO:  REST microservice running on port 9000 single-threaded=0
2021-07-16 13:45:28,526 - seldon_core.microservice:main:486 - INFO:  REST metrics microservice running on port 6000
2021-07-16 13:45:28,527 - seldon_core.microservice:main:496 - INFO:  Starting servers
Traceback (most recent call last):
  File "/Users/minhdang/opt/miniconda3/envs/datascience/bin/seldon-core-microservice", line 8, in <module>
    sys.exit(main())
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/site-packages/seldon_core/microservice.py", line 497, in main
    start_servers(server1_func, server2_func, server3_func, metrics_server_func)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/site-packages/seldon_core/microservice.py", line 64, in start_servers
    p2.start()
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/minhdang/opt/miniconda3/envs/datascience/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.grpc_prediction_server'

Environment

OS: MacOS
Python version: 3.9.5

The text was updated successfully, but these errors were encountered:

ukclivecox · 2021-07-29T17:46:39Z

I see you are using python 3.9 see #1230

Can you try with python 3.7 for your use case?

damosuzuki · 2021-10-01T16:56:27Z

FYI: Reading the linked issue, it seems that Python 3.8 support has been merged, but I am still facing the same error reported above on 3.8.5.

ukclivecox · 2021-10-01T17:18:29Z

Thanks @damosuzuki can you give any further insight into the issue?

damosuzuki · 2021-10-01T17:58:25Z

Not sure I have more insight, but I can be more verbose :)

I am trying to follow the same instructions as @minhdang241 (https://docs.seldon.io/projects/seldon-core/en/v1.1.0/python/python_component.html), and I receive the same error when trying to start the example model with seldon-core-microservice with Python 3.8.5 (macos). I noticed that #1230 adds support for Python 3.8, but I am not sure if this impacts seldon-core-microservice.

damosuzuki · 2021-10-01T21:02:46Z

I see now that seldon-core-microservice is used in s2i which is now on Python 3.8 by default. I wonder if this is a MacOS specific issue.

ukclivecox · 2022-01-09T15:39:20Z

Please reopen if still an issue on MacOS

kfleischmann · 2022-01-17T15:11:35Z

Have the same problem. Can i support to debug this issue?

ukclivecox · 2022-01-17T15:14:03Z

Thanks. Do you have any further logs or insights here @kfleischmann

jeremy-london · 2022-01-19T18:12:26Z

Hitting this error on Python 3.9.9 using MacOS Big Sur 11.6.2

2022-01-19 11:05:30,987 - seldon_core.microservice:main:567 - INFO:  Starting servers
Traceback (most recent call last):
  File "/Users/e346901/Projects/Gitlab/dai/radlab/irad/mlops-irad/workspaces/s92-nlp/env/bin/seldon-core-microservice", line 8, in <module>
    sys.exit(main())
  File "/env/lib/python3.9/site-packages/seldon_core/microservice.py", line 568, in main
    start_servers(server1_func, server2_func, server3_func, metrics_server_func)
  File "/env/lib/python3.9/site-packages/seldon_core/microservice.py", line 72, in start_servers
    p2.start()
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.grpc_prediction_server'

Also seeing this error on Python 3.8.9 using MacOS Big Sur 11.6.2

2022-01-19 13:06:28,726 - seldon_core.microservice:main:567 - INFO:  Starting servers
Traceback (most recent call last):
  File "/env/bin/seldon-core-microservice", line 8, in <module>
    sys.exit(main())
  File "/env/lib/python3.8/site-packages/seldon_core/microservice.py", line 568, in main
    start_servers(server1_func, server2_func, server3_func, metrics_server_func)
  File "/env/lib/python3.8/site-packages/seldon_core/microservice.py", line 72, in start_servers
    p2.start()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.grpc_prediction_server'

simanto604newscred · 2022-01-26T23:56:43Z

Facing same with 3.9

RuiLi-5413 · 2022-03-24T15:28:30Z

Same error with 3.8.6

RafalSkolasinski · 2022-04-19T15:43:07Z

Did anyone manage to reproduce issue in Linux environment?

gerilya · 2022-05-22T14:51:54Z

It looks like it's related to the way multiprocessing is implemented on MacOs and is the same issue as reported here: pydoit/doit#372

I've checked the PR which resolves it and modified microservice.py as follows:
Replaced the following line:
import multiprocess as mp
With the following lines:

try:
        import multiprocess as mp
        HAS_MULTIPROCESS = True
except ImportError:
        import multiprocessing as mp

That fixed the issue for me. I have multiprocess 0.70.12.2 installed.

ukclivecox · 2022-05-22T15:02:21Z

This is great - thanks @gerilya - we can include this in the next release @RafalSkolasinski

RafalSkolasinski · 2022-05-23T09:36:03Z

Nice. However, rather then defaulting to the multiprocess library if it is installed I would suggest to have it driven by an environmental variable.

RafalSkolasinski · 2022-05-23T15:00:19Z

@gerilya We have a PR containing a fix for this: #4114. The behaviour is controlled via environmental variable.
You should be able to set USE_MULTIPROCESS_PACKAGE=true now in order to do local testing on MacOS.

minhdang241 added bug triage Needs to be triaged and prioritised accordingly labels Jul 16, 2021

ukclivecox added awaiting-feedback and removed triage Needs to be triaged and prioritised accordingly labels Jul 29, 2021

marrrcin mentioned this issue Oct 22, 2021

Unable to start almost empty project with seldon-core-microservice on Python 3.8+ #3697

Closed

ukclivecox closed this as completed Jan 9, 2022

ukclivecox reopened this Jan 19, 2022

ukclivecox removed the awaiting-feedback label Jan 19, 2022

ukclivecox assigned RafalSkolasinski and unassigned RafalSkolasinski Apr 24, 2022

RafalSkolasinski self-assigned this May 23, 2022

RafalSkolasinski mentioned this issue May 23, 2022

use alternative multiprocessing library if USE_MULTIPROCESS_PACKAGE i… #4114

Merged

axsaucedo closed this as completed in #4114 May 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't pickle local object 'main.<locals>.grpc_prediction_server' #3410

Can't pickle local object 'main.<locals>.grpc_prediction_server' #3410

minhdang241 commented Jul 16, 2021 •

edited

Loading

ukclivecox commented Jul 29, 2021 •

edited

Loading

damosuzuki commented Oct 1, 2021

ukclivecox commented Oct 1, 2021

damosuzuki commented Oct 1, 2021 •

edited

Loading

damosuzuki commented Oct 1, 2021

ukclivecox commented Jan 9, 2022

kfleischmann commented Jan 17, 2022

ukclivecox commented Jan 17, 2022

jeremy-london commented Jan 19, 2022 •

edited

Loading

simanto604newscred commented Jan 26, 2022

RuiLi-5413 commented Mar 24, 2022

RafalSkolasinski commented Apr 19, 2022

gerilya commented May 22, 2022

ukclivecox commented May 22, 2022

RafalSkolasinski commented May 23, 2022

RafalSkolasinski commented May 23, 2022

Can't pickle local object 'main.<locals>.grpc_prediction_server' #3410

Can't pickle local object 'main.<locals>.grpc_prediction_server' #3410

Comments

minhdang241 commented Jul 16, 2021 • edited Loading

Describe the bug

Environment

ukclivecox commented Jul 29, 2021 • edited Loading

damosuzuki commented Oct 1, 2021

ukclivecox commented Oct 1, 2021

damosuzuki commented Oct 1, 2021 • edited Loading

damosuzuki commented Oct 1, 2021

ukclivecox commented Jan 9, 2022

kfleischmann commented Jan 17, 2022

ukclivecox commented Jan 17, 2022

jeremy-london commented Jan 19, 2022 • edited Loading

simanto604newscred commented Jan 26, 2022

RuiLi-5413 commented Mar 24, 2022

RafalSkolasinski commented Apr 19, 2022

gerilya commented May 22, 2022

ukclivecox commented May 22, 2022

RafalSkolasinski commented May 23, 2022

RafalSkolasinski commented May 23, 2022

minhdang241 commented Jul 16, 2021 •

edited

Loading

ukclivecox commented Jul 29, 2021 •

edited

Loading

damosuzuki commented Oct 1, 2021 •

edited

Loading

jeremy-london commented Jan 19, 2022 •

edited

Loading