Skip to main content
 首页 » 编程设计

python之如何结合 multiprocessing 和 eventlet

2025年01月19日10yxwkf

我有一个任务需要启动 2 个进程,并且在每个进程中需要启动 2 个线程才能真正工作。 下面是我用来模拟用例的源代码。

import multiprocessing 
import eventlet 
 
def subworker(num1, num2): 
    print 'Start subworker %d,%d\n' % (num1, num2) 
    eventlet.sleep(10) 
    print 'End subworker %d,%d\n' % (num1, num2) 
 
def worker(**kwargs): 
    number = kwargs['number'] 
    pool = eventlet.GreenPool(size=2) 
    pool.spawn_n(subworker, number, 1) 
    pool.spawn_n(subworker, number, 2) 
    pool.waitall() 
 
def launcher(number): 
    kwargs = {'number': number} 
    th = multiprocessing.Process(target=worker, kwargs=kwargs) 
    th.start() 
    while True: 
        if not th.is_alive(): 
            break 
        eventlet.sleep(0) 
 
    th.join() 
 
 
def main(): 
    pool = eventlet.GreenPool(size=2) 
    pool.spawn_n(launcher, 1) 
    pool.spawn_n(launcher, 2) 
    pool.waitall() 
 
main() 

当我运行这个 python 脚本时,我的预期输出是这样的:

Start subworker 1,1  
Start subworker 1,2 
Start subworker 2,1 
Start subworker 2,2 
End subworker 1,1 
End subworker 1,2 
End subworker 2,1 
End subworker 2,2

But what I really got is:

Start subworker 1,1 
Traceback (most recent call last): 
 
  File "/Users/leehom/python_local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl 
Start subworker 1,2 
    func(*args, **kwargs) 
 
  File "/Users/leehom/Desktop/home/work_dir/source/snips/Test_multiprocessing_and_eventlet.py", line 27, in launcher 
Start subworker 2,1 
    if not th.is_alive(): 
 
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 155, in is_alive 
Start subworker 2,2 
    assert self._parent_pid == os.getpid(), 'can only test a child process' 
 
AssertionError: can only test a child process 
Start subworker 1,1 
Start subworker 1,2 
Start subworker 2,1 
Start subworker 2,2 
End subworker 1,1 
End subworker 1,2 
End subworker 2,1 
End subworker 2,2 
End subworker 1,1 
End subworker 1,2 
End subworker 2,1 
End subworker 2,2 
Process finished with exit code 0

Seems launcher was called twice. I do not understand why this happen.

If I comment one line in my main function

def main(): 
    pool = eventlet.GreenPool(size=2) 
    pool.spawn_n(launcher, 1) 
    # pool.spawn_n(launcher, 2) 
    pool.waitall() 

结果是对的:

Start subworker 1,1 
Start subworker 1,2 
End subworker 1,1 
End subworker 1,2

有人知道我该如何解决这个问题以及为什么会出现这个问题吗?

请您参考如下方法:

截至 2018-01,Eventlet 和多处理不能很好地协同工作。您最好的选择是在外部生成工作进程。第二个最佳选择是 os.fork() 创建工作进程然后才 import eventlet。

订阅此问题以在多处理兼容性得到解决时收到通知。 https://github.com/eventlet/eventlet/issues/147