Skip to main content
 首页 » 编程设计

用于字符串列表的 Python 2.7 多处理池

2024年10月24日27over140

我是 Python 新手(免责声明:我是编程新手,我已经在线阅读 Python 两周了)并且我编写了一个简单的多处理脚本,应该允许我使用四个子进程一次。我正在使用一个全局变量(是的,我现在知道得更多了)来跟踪同时运行的进程数。开始一个新的进程,加一;结束一个进程,减一。这很困惑,但我只专注于让多进程工作,它确实如此。

到目前为止,我一直在做相当于:

processes = 0 
 
def function(value) 
    global processes 
    do stuff to value 
    processes-=1 
 
While read line 
    if processes < 4 
        processes+=1 
        create a new subprocess - function(line) 

1:我需要以比全局更好的方式跟踪流程。我看到一些在 python 中使用“池”来拥有 4 个 worker ,但我失败了。我喜欢池的想法,但我不知道如何将列表的每一行传递给下一个 worker 。想法?

2:按一般原则,为什么我的全局变量减量不起作用?我知道这很丑陋,但我至少希望它丑陋且成功。

3:我知道在编辑之前我没有锁定 var,我打算在递减正常工作后添加它。

如果那是可怕的伪代码,我很抱歉,但我认为您可以看出要点。如果您想深入了解,这是真正的代码:

MAX_THREADS = 4 
CURRENT_THREADS = 0 
MAX_LOAD = 8 
 
# Iterate through all users in the userlist and call the funWork function on each user 
def funReader(filename): 
 
    # I defined the logger in detail above, I skipped about 200 lines of code to get it slimmed down 
    logger.info("Starting 'move' function for file \"{0}\"...".format(filename)) 
 
    # Read in the entire user list file 
    file = open(filename, 'r') 
    lines = file.read() 
    file.close() 
 
    for line in lines: 
        user = line.rstrip() 
        funControl(user) 
 
# Accept a username and query system load and current funWork thread count; decide when to start next thread 
def funControl(user): 
 
    # Global variables that control whether a new thread starts 
    global MAX_THREADS 
    global CURRENT_THREADS 
    global MAX_LOAD 
 
    # Decide whether to start a new subprocess of funWork for the current user 
    print 
    logger.info("Trying to start a new thread for user {0}".format(user)) 
    sysLoad = os.getloadavg()[1] 
 
    logger.info("The current threads before starting a new loop are: {0}.".format(CURRENT_THREADS)) 
    if CURRENT_THREADS < MAX_THREADS: 
        if sysLoad < MAX_LOAD: 
            CURRENT_THREADS+=1 
            logger.info("Starting a new thread for user {0}.".format(user)) 
            p = Process(target=funWork, args=(user,)) 
            p.start() 
        else: 
            print "Max Load is {0}".format(MAX_LOAD) 
            logger.info("System load is too high ({0}), process delayed for four minutes.".format(sysLoad)) 
            time.sleep(240) 
            funControl(user) 
    else: 
        logger.info("There are already {0} threads running, user {1} delayed for ten minutes.".format(CURRENT_THREADS, user)) 
        time.sleep(600) 
        funControl(user) 
 
# Actually do the work for one user 
def funWork(user): 
 
    global CURRENT_THREADS 
 
    for x in range (0,10): 
        logger.info("Processing user {0}.".format(user)) 
        time.sleep(1) 
    CURRENT_THREADS-=1 

最后:您看到的任何错误都可能是转录错误,因为代码在工作的服务器上执行时没有错误。但是,您看到的任何可怕的编码实践都完全是我的。

提前致谢!

请您参考如下方法:

这个怎么样:(未测试)

MAX_PROCS = 4 
 
# Actually do the work for one user 
def funWork(user): 
 
    for x in range (0,10): 
        logger.info("Processing user {0}.".format(user)) 
        time.sleep(1) 
    return 
 
# Iterate through all users in the userlist and call the funWork function on each user 
def funReader(filename): 
 
    # I defined the logger in detail above, I skipped about 200 lines of code to get it slimmed down 
    logger.info("Starting 'move' function for file \"{0}\"...".format(filename)) 
 
    # Read in the entire user list file 
    file = open(filename, 'r') 
    lines = file.read() 
    file.close() 
 
    work = [] 
    for line in lines: 
        user = line.rstrip() 
        work.append(user) 
    pool = multiprocessing.Pool(processes=MAX_PROCS)  #threads are different from processes... 
    return pool.map(funWork, work)