Skip to main content
 首页 » 编程设计

python之multiprocessing.Manager().Queue() 可以处理复杂的对象吗

2024年10月24日10rubylouvre

def load_cache(cache_name, resultq=None): 
 
    with open(cache_name, 'rb') as input: 
        cached_data = pickle.load(input) 
 
    if not resultq: 
        return cached_data 
    else: 
        try: 
            resultq.put( (cache_name, cached_data) )    # getting error here 
        except: 
            print sys.exc_info()[1] 
 
q = multiprocessing.Manager().Queue() 
info = ['a', 'b', 'c', 'd'] 
procs = [] 
for i in zip(info, itertools.repeat(q)): 
    p = multiprocessing.Process(target=load_cache, args=i) 
    procs.append(p) 
    p.daemon = False 
    p.start() 
 
for p in procs: 
    p.join() 

输出:

Traceback (most recent call last): 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 240, in serve_client 
         request = recv() 
TypeError: __new__() takes at least 2 arguments (1 given) 

不确定此错误的含义或如何解决此问题。有什么想法吗?

注意:
如果我将 cached_data 设置为一些虚拟值,例如 cached_data={},它可以正常工作。
cached_data 是一个包含复杂 Python 对象的字典

如果我删除 resultq.put() 周围的 try/except,

File "main.py" line 20 in load_cache 
    resultq.put((cache_name, cached_data)) 
File "<string>", line 2, in put 
File "/usr/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod 
    raise convert_to_error(kind, result) 
RemoteError: 

请您参考如下方法:

很难从你的问题中判断出发生了什么,因为有一行

q = multiprocessing.Manager().Queue() 

引用一个 Queue,但它是唯一的。目前尚不清楚您的错误源自何处,因为导致错误的行似乎不在您的列表中。


multiprocessing.Queue 将字节从一个地方铲到另一个地方。字节或少或多,但不简单也不复杂。为了通过 Queue 移动对象,它们需要被序列化/反序列化(转换为/从字节形式)。 众所周知,pickle(和 cPickle)在序列化方面存在问题。 dill 你可能会好运(我对此有积极的经验)。来自 dill 文档:

Dill extends python’s ‘pickle’ module for serializing and de-serializing python objects to the majority of the built-in python types. ... Dill provides the user the same interface as the ‘pickle’ module, and also includes some additional features. In addition to pickling python objects, dill provides the ability to save the state of an interpreter session in a single command... Dill can be used to store python objects to a file, but the primary usage is to send python objects across the network as a byte stream.