python之在 Flask 应用程序中开始使用 Celery 时出现 PicklingError

我希望使用 Celery/RabbitMQ 在 Windows 7 和 Python 3.3 上的 Flask 应用程序中运行常规间隔任务。 Pip 安装:

billiard==3.3.0.10 
celery==3.1.6 

代码:

### celeryapp.py ### 
from celery import Celery 
from twend import config, app 
 
def make_celery(app): 
    celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL']) 
    celery.conf.update(app.config) 
    TaskBase = celery.Task 
    class ContextTask(TaskBase): 
        abstract = True 
        def __call__(self, *args, **kwargs): 
            with app.app_context(): 
                return TaskBase.__call__(self, *args, **kwargs) 
    celery.Task = ContextTask 
    return celery 
 
celery_pipe = make_celery(app) 
celery_pipe.config_from_object(config) 
 
### tasks.py ### 
from celery import task 
from twend.celeryapp import celery_pipe as celery 
 
@celery.task() 
def add_together(a, b): 
    return a + b 
 
### config.py ### 
CELERY_BROKER_URL = 'amqp://' 
CELERY_RESULT_BACKEND = 'amqp://' 
CELERY_ACCEPT_CONTENT = ['json'] 

当我尝试启动 worker 时，我得到:

ERROR/MainProcess] Unrecoverable error: PicklingError( 
"Can't pickle <class 'module'>: attribute lookup builtins.module failed",) 
Traceback (most recent call last): 
  File "c:\Python33\twend\lib\site-packages\celery\worker\__init__.py", line 212 
, in start 
    self.blueprint.start(self) 
  File "c:\Python33\twend\lib\site-packages\celery\bootsteps.py", line 123, in s 
tart 
    step.start(parent) 
  File "c:\Python33\twend\lib\site-packages\celery\bootsteps.py", line 373, in s 
tart 
    return self.obj.start() 
  File "c:\Python33\twend\lib\site-packages\celery\concurrency\base.py", line 12 
7, in start 
    self.on_start() 
  File "c:\Python33\twend\lib\site-packages\celery\concurrency\prefork.py", line 
 112, in on_start 
    **self.options) 
  File "c:\Python33\twend\lib\site-packages\billiard\pool.py", line 966, in __in 
it__ 
    self._create_worker_process(i) 
  File "c:\Python33\twend\lib\site-packages\billiard\pool.py", line 1059, in _cr 
eate_worker_process 
    w.start() 
  File "c:\Python33\twend\lib\site-packages\billiard\process.py", line 139, in s 
tart 
    self._popen = Popen(self) 
  File "c:\Python33\twend\lib\site-packages\billiard\forking.py", line 263, in _ 
_init__ 
    dump(process_obj, to_child, HIGHEST_PROTOCOL) 
  File "c:\Python33\twend\lib\site-packages\billiard\_reduction3.py", line 60, i 
n dump 
    ForkingPickler(file, protocol).dump(obj) 
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup builtins. 
module failed 
 
(twend) c:\Python33\twend>Traceback (most recent call last): 
  File "<string>", line 1, in <module> 
  File "c:\Python33\twend\lib\site-packages\billiard\forking.py", line 457, in m 
ain 
    self = load(from_parent) 
EOFError 

这https://github.com/celery/django-celery/issues/228似乎是一个类似的问题，但已在我使用的 Celery 版本中修复。

我还读到 Windows 对 pickle 有限制，但我不确定实际被 pickle 的是什么。

请您参考如下方法:

Celery 使用酸洗来序列化任务和结果。我看到您已经将配置设置为仅接受 json 序列化，因此也许您可以通过添加更多配置行来解决您的问题:

CELERY_RESULT_SERIALIZER = 'json' 
CELERY_TASK_SERIALIZER = 'json' 

这样就完全不用pickle了，Celery的序列化全部使用json来完成。这在理论上会导致性能下降，但我通过这种方式解决了我的 Flask 应用程序中的类似问题并且没有发现任何性能问题。 YMMV