Skip to main content
 首页 » 编程设计

python之如何在生成的线程中捕获内存错误

2024年10月25日9zfyouxi

我以前从未使用过多处理库,所以欢迎所有建议..

我有一个 python 程序,它使用 multiprocessing 库在多个进程中执行一些内存密集型任务,偶尔会耗尽内存(我正在进行优化,但这不是这个问题的目的)。有时,会以一种我似乎无法捕获的方式抛出内存不足错误(下面的输出),然后程序卡在 pool.join() 上(我正在使用multiprocessing.Pool。发生此问题时,如何让程序做一些事情而不是无限期地等待?

理想情况下,内存错误传播回主进程,然后主进程死亡。

这是内存错误:

Exception in thread Thread-1: 
Traceback (most recent call last): 
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner 
    self.run() 
  File "/usr/lib64/python2.7/threading.py", line 764, in run 
    self.__target(*self.__args, **self.__kwargs) 
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 325, in _handle_workers 
    pool._maintain_pool() 
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 229, in _maintain_pool 
    self._repopulate_pool() 
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool 
    w.start() 
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start 
    self._popen = Popen(self) 
  File "/usr/lib64/python2.7/multiprocessing/forking.py", line 121, in __init__ 
    self.pid = os.fork() 
OSError: [Errno 12] Cannot allocate memory  

这是我管理多处理的地方:

mp_pool = mp.Pool(processes=num_processes) 
mp_results = list() 
for datum in input_data: 
    data_args = { 
         'value': 0 // actually some other simple dict key/values 
    } 
    mp_results.append(mp_pool.apply_async(_process_data, args=(common_args, data_args))) 
frame_pool.close() 
frame_pool.join()  // hangs here when that thread dies.. 
for result_async in mp_results: 
    result = result_async.get() 
    // do stuff to collect results 
// rest of the code 

当我中断挂起的程序时,我得到:

Process process_003: 
Traceback (most recent call last): 
  File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap 
    self.run() 
  File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run 
    self._target(*self._args, **self._kwargs) 
  File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/pool.py", line 102, in worker 
    task = get() 
  File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/queues.py", line 374, in get 
    return recv() 
    racquire() 
KeyboardInterrupt 

请您参考如下方法:

这实际上是一个known bug in python's multiprocessing module ,在 python 3 ( here's a summarizing blog post I found ) 中修复。 python issue 22393 附有补丁,但尚未正式应用。

基本上,如果多进程池的子进程之一意外死亡(内存不足、外部终止等),该池将无限期等待。