python之多处理脚本比普通脚本慢

我似乎无法全神贯注于多处理。我正在尝试做一些基本操作，但多处理脚本似乎需要永远。

import multiprocessing, time, psycopg2 
 
class Consumer(multiprocessing.Process): 
 
    def __init__(self, task_queue, result_queue): 
        multiprocessing.Process.__init__(self) 
        self.task_queue = task_queue 
        self.result_queue = result_queue 
 
    def run(self): 
        proc_name = self.name 
        while True: 
            next_task = self.task_queue.get() 
            if next_task is None: 
                print ('Tasks Complete') 
                self.task_queue.task_done() 
                break             
            answer = next_task() 
            self.task_queue.task_done() 
            self.result_queue.put(answer) 
        return 
 
class Task(object): 
    def __init__(self, a): 
        self.a = a 
 
    def __call__(self): 
        #Some more work will go in here but for now just return the value 
        return self.a 
 
    def __str__(self): 
        return 'ARC' 
    def run(self): 
        print ('IN') 
 
 
if __name__ == '__main__': 
    start_time = time.time() 
    numberList = [] 
 
    for x in range(1000000): 
        numberList.append(x)  
 
    result = [] 
    counter = 0 
    total = 0 
    for id in numberList: 
        total =+ id 
        counter += 1 
    print(counter) 
    print("Finished in Seconds: %s" %(time.time()-start_time)) 
    ############################################################################################################################### 
    #Mutliprocessing starts here.... 
    ###############################################################################################################################         
    start_time = time.time() 
    tasks = multiprocessing.JoinableQueue() 
    results = multiprocessing.Queue() 
 
    num_consumers = multiprocessing.cpu_count()  
    consumers = [Consumer(tasks, results) for i in range(num_consumers)] 
    for w in consumers: 
        w.start() 
 
    num_jobs = len(numberList) 
 
    for i in range(num_jobs): 
        tasks.put(Task(numberList[i])) 
 
    for i in range(num_consumers): 
        tasks.put(None) 
 
    print("So far: %s" %(time.time()-start_time)) 
    result = [] 
    while num_jobs: 
        result.append(results.get()) 
        num_jobs -= 1 
    print (len(result)) 
    print("Finished in Seconds: %s" %(time.time()-start_time)) 

原始脚本来自 here

第一个基本的 for 循环平均完成 0.4 秒，多处理循环在 56 秒内完成，而我本以为它是反之亦然..？

是否缺少某些逻辑或者它实际上更慢？否则我将如何构建它以使其比标准 for 循环更快？

请您参考如下方法:

您的多处理代码确实设计过度，实际上并没有完成它应该做的工作。我重写了它以使其更简单，实际上做了它应该做的，现在它比简单的循环更快:

import multiprocessing 
import time 
 
 
def add_list(l): 
    total = 0  
    counter = 0  
    for ent in l: 
        total += ent  
        counter += 1 
    return (total, counter) 
 
def split_list(l, n):  
    # Split `l` into `n` equal lists. 
    # Borrowed from http://stackoverflow.com/a/2136090/2073595 
    return [l[i::n] for i in xrange(n)] 
 
if __name__ == '__main__': 
    start_time = time.time() 
    numberList = range(1000000): 
 
    counter = 0  
    total = 0  
    for id in numberList: 
        total += id 
        counter += 1 
    print(counter) 
    print(total) 
    print("Finished in Seconds: %s" %(time.time()-start_time)) 
    start_time = time.time() 
 
    num_consumers = multiprocessing.cpu_count()  
    # Split the list up so that each consumer can add up a subsection of the list. 
    lists = split_list(numberList, num_consumers) 
    p = multiprocessing.Pool(num_consumers) 
    results = p.map(add_list, lists) 
    total = 0  
    counter = 0  
    # Combine the results each worker returned. 
    for t, c in results: 
        total += t 
        counter += c 
    print(counter) 
    print(total) 
 
    print("Finished in Seconds: %s" %(time.time()-start_time)) 

这是输出:

Standard: 
1000000 
499999500000 
Finished in Seconds: 0.272150039673 
Multiprocessing: 
1000000 
499999500000 
Finished in Seconds: 0.238755941391 

正如@aruisdante 指出的那样，您的工作负载非常轻，因此在这里并没有真正充分感受到多处理的好处。如果您进行更繁重的处理，您会看到更大的差异。