我是 Python 新手(免责声明:我是编程新手,我已经在线阅读 Python 两周了)并且我编写了一个简单的多处理脚本,应该允许我使用四个子进程一次。我正在使用一个全局变量(是的,我现在知道得更多了)来跟踪同时运行的进程数。开始一个新的进程,加一;结束一个进程,减一。这很困惑,但我只专注于让多进程工作,它确实如此。
到目前为止,我一直在做相当于:
processes = 0
def function(value)
global processes
do stuff to value
processes-=1
While read line
if processes < 4
processes+=1
create a new subprocess - function(line)
1:我需要以比全局更好的方式跟踪流程。我看到一些在 python 中使用“池”来拥有 4 个 worker ,但我失败了。我喜欢池的想法,但我不知道如何将列表的每一行传递给下一个 worker 。想法?
2:按一般原则,为什么我的全局变量减量不起作用?我知道这很丑陋,但我至少希望它丑陋且成功。
3:我知道在编辑之前我没有锁定 var,我打算在递减正常工作后添加它。
如果那是可怕的伪代码,我很抱歉,但我认为您可以看出要点。如果您想深入了解,这是真正的代码:
MAX_THREADS = 4
CURRENT_THREADS = 0
MAX_LOAD = 8
# Iterate through all users in the userlist and call the funWork function on each user
def funReader(filename):
# I defined the logger in detail above, I skipped about 200 lines of code to get it slimmed down
logger.info("Starting 'move' function for file \"{0}\"...".format(filename))
# Read in the entire user list file
file = open(filename, 'r')
lines = file.read()
file.close()
for line in lines:
user = line.rstrip()
funControl(user)
# Accept a username and query system load and current funWork thread count; decide when to start next thread
def funControl(user):
# Global variables that control whether a new thread starts
global MAX_THREADS
global CURRENT_THREADS
global MAX_LOAD
# Decide whether to start a new subprocess of funWork for the current user
print
logger.info("Trying to start a new thread for user {0}".format(user))
sysLoad = os.getloadavg()[1]
logger.info("The current threads before starting a new loop are: {0}.".format(CURRENT_THREADS))
if CURRENT_THREADS < MAX_THREADS:
if sysLoad < MAX_LOAD:
CURRENT_THREADS+=1
logger.info("Starting a new thread for user {0}.".format(user))
p = Process(target=funWork, args=(user,))
p.start()
else:
print "Max Load is {0}".format(MAX_LOAD)
logger.info("System load is too high ({0}), process delayed for four minutes.".format(sysLoad))
time.sleep(240)
funControl(user)
else:
logger.info("There are already {0} threads running, user {1} delayed for ten minutes.".format(CURRENT_THREADS, user))
time.sleep(600)
funControl(user)
# Actually do the work for one user
def funWork(user):
global CURRENT_THREADS
for x in range (0,10):
logger.info("Processing user {0}.".format(user))
time.sleep(1)
CURRENT_THREADS-=1
最后:您看到的任何错误都可能是转录错误,因为代码在工作的服务器上执行时没有错误。但是,您看到的任何可怕的编码实践都完全是我的。
提前致谢!
请您参考如下方法:
这个怎么样:(未测试)
MAX_PROCS = 4
# Actually do the work for one user
def funWork(user):
for x in range (0,10):
logger.info("Processing user {0}.".format(user))
time.sleep(1)
return
# Iterate through all users in the userlist and call the funWork function on each user
def funReader(filename):
# I defined the logger in detail above, I skipped about 200 lines of code to get it slimmed down
logger.info("Starting 'move' function for file \"{0}\"...".format(filename))
# Read in the entire user list file
file = open(filename, 'r')
lines = file.read()
file.close()
work = []
for line in lines:
user = line.rstrip()
work.append(user)
pool = multiprocessing.Pool(processes=MAX_PROCS) #threads are different from processes...
return pool.map(funWork, work)
