Skip to main content
 首页 » 编程设计

python之生产者消费者在 Python 中使用信号量和互斥量

2024年11月01日11sharpest

我正在尝试了解如何使用 Python 信号量实现一个缓冲区大小有限的队列,该队列可供多个生产者和消费者使用。这是我的实现:

class Q: 
 
    def __init__(self, size): 
        self.buff = [None]*size 
        self.end = 0 
        self.start = 0 
        self.size = size 
        self.end_lock = Lock()  # protect end from race across multiple producers 
        self.start_lock = Lock()  # protect start from race across multiple consumers 
        self.open = Semaphore(size)  # block till there's space to produce 
        self.closed = Semaphore(size) # block till there's item to consume 
        for _ in range(size):  # initialize with all closed acquired so that consumer is blocked 
            self.closed.acquire() 
 
    def put(self, val): 
        self.open.acquire() 
        with self.end_lock: 
            self.buff[self.end] = val 
            self.end = (self.end+1)%self.size 
        self.closed.release() 
 
    def get(self): 
        self.closed.acquire() 
        with self.start_lock: 
            val = self.buff[(self.start)%self.size] 
            self.start = (self.start+1)%self.size 
        self.open.release() 
        return val 

这个实现没有错误吗?这可以进一步简化以使用更少的互斥锁/信号量吗?

请您参考如下方法:

我觉得不错。信号量防止并发生产者和消费者写入和读取过多,锁防止并发生产者或消费者同时修改 end 或 start 索引。

这两个信号量肯定是必须的。您可以移除其中一把锁并在 get 和 put 中使用它来保护 start 和 end 不允许消费者和生产者同时访问队列的索引。 (CPython's queue implementation 这样做。)


虽然我会删除 size 属性以支持 len(self.buff) 并重命名 start 和 end 分别指向 read_index 和 write_index 的索引(以及锁)。另外,我认为您可以在不持有锁的情况下访问缓冲区(因为 lists themselves are thread-safe ):

    def put(self, val): 
        self.open.acquire() 
        with self.write_lock: 
            index = self.write_index 
            self.write_index = (self.write_index + 1) % len(self.buff) 
        self.buff[index] = val 
        self.closed.release() 
 
    def get(self): 
        self.closed.acquire() 
        with self.read_lock: 
            index = self.read_index 
            self.read_index = (self.read_index + 1) % len(self.buff) 
        val = self.buff[index] 
        self.open.release() 
        return val 

这是我曾经玩过的一个小测试程序:

def producer(queue, start, end, step): 
    for value in range(start, end, step): 
        queue.put(value) 
    print('Producer finished') 
 
 
def consumer(queue, count, result, lock): 
    local_result = [] 
    for _ in range(count): 
        local_result.append(queue.get()) 
    with lock: 
        result.update(local_result) 
    print('Consumer finished') 
 
 
def main(): 
    value_count = 500000 
    producer_count = 50 
    consumer_count = 50 
    assert value_count % producer_count == 0 
    assert value_count % consumer_count == 0 
 
    queue = Queue(123) 
    result = set() 
    lock = Lock() 
    producers = [Thread(target=producer, args=(queue, i, value_count, producer_count)) for i in range(producer_count)] 
    consumers = [Thread(target=consumer, args=(queue, value_count // consumer_count, result, lock)) for _ in range(consumer_count)] 
 
    for p in producers: 
        p.start() 
    for c in consumers: 
        c.start() 
 
    for p in producers: 
        p.join() 
    for c in consumers: 
        c.join() 
 
    if len(result) != value_count: 
        raise ValueError('Result size is %d instead of %d' % (len(result), value_count)) 
 
 
if __name__ == '__main__': 
    main()