Skip to main content
 首页 » 编程设计

python之如何提高 python 循环速度

2025年02月15日12daizhj

我有一个包含 370k 条记录的数据集,存储在需要集成的 Pandas Dataframe 中。我尝试了多处理、线程、CPython 和循环展开。但我没有成功,计算时间显示为 22 小时。任务如下:

%matplotlib inline   
from numba import jit, autojit 
import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt 
 
with open('data/full_text.txt', encoding = "ISO-8859-1") as f: 
 strdata=f.readlines() 
data=[] 
 
for string in strdata: 
 data.append(string.split('\t')) 
 
df=pd.DataFrame(data,columns=["uname","date","UT","lat","long","msg"]) 
 
df=df.drop('UT',axis=1) 
 
df[['lat','long']] = df[['lat','long']].apply(pd.to_numeric) 
 
from textblob import TextBlob 
from tqdm import tqdm 
 
df['polarity']=np.zeros(len(df)) 

线程:

 from queue import Queue 
 from threading import Thread 
 import logging 
 logging.basicConfig( 
 level=logging.DEBUG, 
  format='(%(threadName)-10s) %(message)s', 
  ) 
 
 
class DownloadWorker(Thread): 
   def __init__(self, queue): 
       Thread.__init__(self) 
       self.queue = queue 
 
   def run(self): 
       while True: 
           # Get the work from the queue and expand the tuple 
         lowIndex, highIndex = self.queue.get() 
         a = range(lowIndex,highIndex-1) 
         for i in a: 
            df['polarity'][i]=TextBlob(df['msg'][i]).sentiment.polarity 
         self.queue.task_done() 
 
  def main(): 
   # Create a queue to communicate with the worker threads 
   queue = Queue() 
   # Create 8 worker threads 
   for x in range(8): 
     worker = DownloadWorker(queue) 
     worker.daemon = True 
     worker.start() 
  # Put the tasks into the queue as a tuple 
   for i in tqdm(range(0,len(df)-1,62936)): 
     logging.debug('Queueing') 
     queue.put((i,i+62936 )) 
     queue.join() 
     print('Took {}'.format(time() - ts)) 
 
 main() 

带循环展开的多处理:

pool = multiprocessing.Pool(processes=2) 
r = pool.map(assign_polarity, df) 
pool.close() 
 
def assign_polarity(df): 
   a=range(0,len(df),5) 
   for i in tqdm(a): 
       df['polarity'][i]=TextBlob(df['msg'][i]).sentiment.polarity 
       df['polarity'][i+1]=TextBlob(df['msg'][i+1]).sentiment.polarity 
       df['polarity'][i+2]=TextBlob(df['msg'][i+2]).sentiment.polarity 
       df['polarity'][i+3]=TextBlob(df['msg'][i+3]).sentiment.polarity 
       df['polarity'][i+4]=TextBlob(df['msg'][i+4]).sentiment.polarity 

如何提高计算速度?或者以更快的方式将计算存储在数据框中?我的笔记本配置

  • 内存:8GB
  • 物理内核:2
  • 逻辑核心:8 个
  • Windows 10

实现多处理给了我更多的计算时间。 线程是按顺序执行的(我想是因为 GIL) Loop Unrolling 给了我相同的计算速度。 Cpython 在导入库时给我错误。

请您参考如下方法:

ASD——我注意到在 df 中迭代存储一些东西非常慢。我会尝试将您的 TextBlob 存储在列表(或其他结构)中,然后将该列表转换为 df 的列。