def load_cache(cache_name, resultq=None):
with open(cache_name, 'rb') as input:
cached_data = pickle.load(input)
if not resultq:
return cached_data
else:
try:
resultq.put( (cache_name, cached_data) ) # getting error here
except:
print sys.exc_info()[1]
q = multiprocessing.Manager().Queue()
info = ['a', 'b', 'c', 'd']
procs = []
for i in zip(info, itertools.repeat(q)):
p = multiprocessing.Process(target=load_cache, args=i)
procs.append(p)
p.daemon = False
p.start()
for p in procs:
p.join()
输出:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/managers.py", line 240, in serve_client
request = recv()
TypeError: __new__() takes at least 2 arguments (1 given)
不确定此错误的含义或如何解决此问题。有什么想法吗?
注意:
如果我将 cached_data 设置为一些虚拟值,例如 cached_data={},它可以正常工作。
cached_data 是一个包含复杂 Python 对象的字典
如果我删除 resultq.put() 周围的 try/except,
File "main.py" line 20 in load_cache
resultq.put((cache_name, cached_data))
File "<string>", line 2, in put
File "/usr/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
请您参考如下方法:
很难从你的问题中判断出发生了什么,因为有一行
q = multiprocessing.Manager().Queue()
引用一个 Queue,但它是唯一的。目前尚不清楚您的错误源自何处,因为导致错误的行似乎不在您的列表中。
multiprocessing.Queue 将字节从一个地方铲到另一个地方。字节或少或多,但不简单也不复杂。为了通过 Queue 移动对象,它们需要被序列化/反序列化(转换为/从字节形式)。 众所周知,pickle(和 cPickle)在序列化方面存在问题。 dill 你可能会好运(我对此有积极的经验)。来自 dill 文档:
Dill extends python’s ‘pickle’ module for serializing and de-serializing python objects to the majority of the built-in python types. ... Dill provides the user the same interface as the ‘pickle’ module, and also includes some additional features. In addition to pickling python objects, dill provides the ability to save the state of an interpreter session in a single command... Dill can be used to store python objects to a file, but the primary usage is to send python objects across the network as a byte stream.
