Skip to main content
 首页 » 编程设计

python之celery 即使允许也不会接受 pickle

2024年12月31日8lvdongjie

我正在尝试编写一个将 numpy 数组(或任何任意对象)传递给工作人员的 celery 应用程序。据我所知,这需要通过 pickle 进行序列化(注意:我知道安全隐患,但在这种情况下这不是问题)。

然而,即使在尝试了所有可能的方法之后,我仍然可以找到允许 pickle 作为序列化程序的方法,但我仍然收到以下 kombu 异常:

kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted 
content of type pickle (application/x-python-serialize) 

我当前的文件是:

# tasks.py 
from celery import Celery 
app = Celery( 
    'tasks', 
    broker='redis://localhost', 
    accept_content=['pickle'], 
    task_serializer='pickle' 
) 
 
@app.task 
def adding(x, y): 
    return x + y 
 
if __name__ == '__main__': 
    import numpy as np 
    adding.apply_async((np.array([1]), np.array([1])), serializer='pickle') 

另外我还有一个配置文件:

# celeryconfig.py 
print('configuring...') 
 
accept_content = ['pickle', 'application/x-python-serialize'] 
task_serializer = 'pickle' 
result_serializer = 'pickle' 
from kombu import serialization 
serialization.register_pickle() 
serialization.enable_insecure_serializers() 

但是,如果我运行 worker (celery -A tasks worker --loglevel=info),然后执行进行异步调用的代码 (python tasks.py >),我得到以下回溯。我错过了什么吗?

[2018-06-16 11:46:23,617: CRITICAL/MainProcess] Unrecoverable error: ContentDisallowed('Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)',) 
Traceback (most recent call last): 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start 
    self.blueprint.start(self) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start 
    step.start(parent) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start 
    return self.obj.start() 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start 
    blueprint.start(self) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start 
    step.start(parent) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/consufrom celery import Celery 
mer/consumer.py", line 598, in start 
    c.loop(*c.loop_args()) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop 
    next(loop) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop 
    cb(*cbargs) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/redis.py", line 1040, in on_readable 
    self.cycle.on_readable(fileno) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/redis.py", line 337, in on_readable 
    chan.handlers[type]() 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/redis.py", line 724, in _brpop_read 
    self.connection._deliver(loads(bytes_to_str(item)), dest) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver 
    callback(message) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 633, in _callback 
    return callback(message) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback 
    return on_m(message) if on_m else self.receive(decoded, message) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 572, in on_task_received 
    callbacks, 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/celery/worker/strategy.py", line 136, in task_message_handler 
    if body is None and 'args' not in message.payload: 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/message.py", line 207, in payload 
    return self._decoded_cache if self._decoded_cache else self.decode() 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/message.py", line 192, in decode 
    self._decoded_cache = self._decode() 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/message.py", line 197, in _decode 
    self.content_encoding, accept=self.accept) 
  File "/opt/anaconda/envs/Python3/lib/python3.6/site-packages/kombu/serialization.py", line 253, in loads 
    raise self._for_untrusted_content(content_type, 'untrusted') 
kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize) 

请您参考如下方法:

对于任何回答这个问题的人:

答案是使用 app.config_from_object 方法:

import celeryconfig 
app.config_from_object(celeryconfig)