Skip to main content
 首页 » 编程设计

Python 2.7 ProcessPoolExecutor 抛出 IOError : [Errno 32] Broken pipe

2025年02月15日10JustinYoung

我正在将数据以 block 的形式流式传输到一个类中。对于每个数据 block ,两种不同类型的 np.convolve() 在同一个 ProcessPoolExecutor 上执行。调用的卷积类型由返回变量确定。 必须维护数据的顺序,因此每个 future 都有一个关联的序列号。 output 函数强制只返回来自连续 future 的数据(下面未显示)。据我了解,我正在正确调用 ProcessPoolExecutor.shutdown() 函数,但我仍然收到 IOError:

错误是:

$ python processpoolerror.py  
ran 5000000 samples in 3.70395112038 sec: 1.34990982265 Msps 
Traceback (most recent call last): 
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 268, in _feed 
    send(obj) 
IOError: [Errno 32] Broken pipe 

抱歉,它有点长,但我在保持错误的同时尽可能地减少了这个类。在我的机器 Ubuntu 16.04.2 和 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz 上,配对的向下代码总是给出这个错误。在此代码的非修剪版本中,Broken pipe 发生的概率为 25%。

如果您将 78 行编辑为 True,并在执行期间打印,则不会抛出错误。如果您减少 100 行的数据量,则不会引发错误。我在这里做错了什么?谢谢。

import numpy as np 
from concurrent.futures import ProcessPoolExecutor 
import time 
 
 
def _do_xcorr3(rev_header, packet_chunk, seq): 
    r1 = np.convolve(rev_header, packet_chunk, 'full') 
    return 0, seq, r1 
 
def _do_power3(power_kernel, packet_chunk, seq): 
    cp = np.convolve(power_kernel, np.abs(packet_chunk) ** 2, 'full') 
    return 1, seq, cp 
 
class ProcessPoolIssues(): 
 
    ## Constructor 
    # @param chunk_size how many samples to feed in during input() stage 
    def __init__(self,header,chunk_size=500,poolsize=5): 
        self.chunk_size = chunk_size  ##! How many samples to feed 
 
        # ProcessPool stuff 
        self.poolsize = poolsize 
        self.pool = ProcessPoolExecutor(poolsize) 
        self.futures = [] 
 
        # xcr stage stuff 
        self.results0 = [] 
        self.results0.append((0, -1, np.zeros(chunk_size))) 
 
        # power stage stuff 
        self.results1 = [] 
        self.results1.append((1, -1, np.zeros(chunk_size))) 
 
        self.countin = 0 
        self.countout = -1 
 
    def shutdown(self): 
        self.pool.shutdown(wait=True) 
 
 
    ## Returns True if all data has been extracted for given inputs 
    def all_done(self): 
        return self.countin == self.countout+1 
 
    ## main function 
    # @param packet_chunk an array of chunk_size samples to be computed 
    def input(self, packet_chunk): 
        assert len(packet_chunk) == self.chunk_size 
 
        fut0 = self.pool.submit(_do_xcorr3, packet_chunk, packet_chunk, self.countin) 
        self.futures.append(fut0) 
 
        fut1 = self.pool.submit(_do_power3, packet_chunk, packet_chunk, self.countin) 
        self.futures.append(fut1) 
 
        self.countin += 1 
 
 
    # loops through thread pool, copying any results from done threads into results0/1 (and then terminating them) 
    def cultivate_pool(self): 
        todel = [] 
 
        for i, f in enumerate(self.futures): 
            # print "checking", f 
            if f.done(): 
                a, b, c = f.result() 
                if a == 0: 
                    self.results0.append((a,b,c))  # results from one type of future 
                elif a == 1: 
                    self.results1.append((a,b,c))  # results from another type of future 
                todel.append(i) 
 
        # now we need to remove items from futures that are done 
        # we need do it in reverse order so we remove items from the end first (thereby not affecting indices as we go) 
        for i in sorted(todel, reverse=True): 
            del self.futures[i] 
 
            if False:  # change this to true and error goes away 
                print "deleting future #", i 
 
    # may return None 
    def output(self): 
 
        self.cultivate_pool()  # modifies self.results list 
 
        # wait for both results to be done before clearing 
        if len(self.results0) and len(self.results1): 
            del self.results0[0] 
            del self.results1[0] 
            self.countout += 1 
 
        return None 
 
 
def testRate(): 
    chunk = 500 
 
    # a value of 10000 will throw:  IOError: [Errno 32] Broken pipe 
    # smaller values like 1000 do not 
    din = chunk * 10000 
 
    np.random.seed(666) 
    search = np.random.random(233) + np.random.random(233) * 1j 
    input = np.random.random(din) + np.random.random(din) * 1j 
 
    pct = ProcessPoolIssues(search, chunk, poolsize=8) 
 
    st = time.time() 
    for x in range(0, len(input), chunk): 
        slice = input[x:x + chunk] 
        if len(slice) != chunk: 
            break 
        pct.input(slice) 
        pct.output() 
 
    while not pct.all_done(): 
        pct.output() 
 
    ed = time.time() 
    dt = ed - st 
 
    print "ran", din, "samples in", dt, "sec:", din / dt / 1E6, "Msps" 
 
    pct.shutdown() 
 
if __name__ == '__main__': 
    testRate() 

请您参考如下方法:

这可能是因为当您尝试一次发送较大的数据 block 时超出了管道的缓冲区大小。

def _do_xcorr3(rev_header, packet_chunk, seq): 
    r1 = np.convolve(rev_header, packet_chunk, 'full') 
    return 0, seq, r1 
 
def _do_power3(power_kernel, packet_chunk, seq): 
    cp = np.convolve(power_kernel, np.abs(packet_chunk) ** 2, 'full') 
    return 1, seq, cp 

r1 和 cp 的值非常大,因为您正在与 block 的平方进行卷积。

因此,当您尝试以更大的 block 大小运行它时,IO 管道的缓冲区无法处理它。引用this以便更清楚地理解。

至于问题的第二部分,

if False:  # change this to true and error goes away 
                print "deleting future #", i 

在 py3 文档中找到:

16.2.4.4. Reentrancy Binary buffered objects (instances of BufferedReader, BufferedWriter, BufferedRandom and BufferedRWPair) are not reentrant. While reentrant calls will not happen in normal situations, they can arise from doing I/O in a signal handler. If a thread tries to re-enter a buffered object which it is already accessing, a RuntimeError is raised. Note this doesn’t prohibit a different thread from entering the buffered object. The above implicitly extends to text files, since the open() function will wrap a buffered object inside a TextIOWrapper. This includes standard streams and therefore affects the built-in function print() as well.